Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Ye, I noticed - I just relied on the documentation in this case. Andrey On 11/18/19 3:01 PM, Christian König wrote: Well then we should probably update the documentation. Take a look at the implementation, there is no compiler or SMP barrier at all. Christian. Am 18.11.19 um 18:01 schrieb Andrey Grodzovsky: The documentation states it can be used safely with concurrent list_del_init so I assume it's true - but I think my even bigger mistake is that without locking i just do list_first_entry right after list_empty_careful and by this can grab pointer to the same job as concurrent drm_sched_job_timedout->list_first_entry_or_null - so yes I see now i have to use locking as you advised there and then i don't need the list_empty_careful. Andrey On 11/18/19 11:44 AM, Christian König wrote: list_empty_careful() should only be used for optimizing cases, but never if you need to rely on the result. The problem is that the function doesn't has any memory barriers whatsoever, it just checks if the next and prev pointer are both empty instead of just the next pointer. Christian. Am 18.11.19 um 17:23 schrieb Andrey Grodzovsky: Can you explain why ? As I see it - list_empty_careful is specifically designed for the case where the only other concurrent operation in progress is list_del_init (https://www.kernel.org/doc/htmldocs/kernel-api/API-list-empty-careful.html) - which is exactly what happens in this patch, no other list altering operation can take place concurrently - so it looks safe to use for me. Andrey On 11/18/19 11:16 AM, Christian König wrote: Hi Andrey, the only thing which doesn't looks so good is the switch to list_empty_careful in drm_sched_cleanup_jobs. We either take the lock here or we don't, but please not that extra checking. Christian. Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky: Thanks Emily. Christan - ping for review. Andrey On 11/14/19 11:39 PM, Deng, Emily wrote: Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Friday, November 15, 2019 6:14 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Attached. Emily - can you give it a try ? Andrey On 11/14/19 3:12 AM, Christian König wrote: What about instead of peeking at the job to actually remove it from ring_mirror_list right there, Also an interesting idea. We would need to protect the mirror list with a lock again, but that should be the lesser evil. Maybe prototype that and see if it works or not. Regards, Christian. Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: On 11/13/19 9:20 AM, Christian König wrote: Another more fundamental question: Could we get rid of the timeout job at all? There are other stuff there besides picking the first unfinished job which is common for all the drivers - such as freeing guilty signaled job and rearming the timeout work timer. I mean we used to give this as parameter to the scheduler callback because we had the timeout worker in the job, but that is no longer the case. E.g. in drm_sched_job_timedout() we do the following: job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); Why don't we just remove that here and only get the first job after we have stopped the scheduler? Should be ok since we have the extra check for __kthread_should_park in drm_sched_cleanup_jobs which will protect us in this case from a wakeup of sched thread and execution of in drm_sched_cleanup_jobs after we already parked it. The problem here is we need the drm_sched_job to access the private data for each client driver (see amdgpu_job_timedout for example). What about instead of peeking at the job to actually remove it from ring_mirror_list right there, go ahead with it through the reset routine, if it's signaled in the meanwhile that great - release it, otherwise put it back into ring_mirror_list in drm_sched_resubmit_jobs. Andrey Regards, Christian. Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: This why I asked for a trace with timer enabled, but since there is a finite number of places we touch the timer Emily can just put prints there. Also, I wonder if this temp fix helps her with the issue or not. Andrey On 11/13/19 2:36 AM, Christian König wrote: The question is where do we rearm the timer for this problem to occur? Regards, Christian. Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: I was able to reproduce the crash by using the attached simulate_crash.patch - waiting on guilty job to signal in reset work and artificially rearming the timeout timer just before the check for !cancel_delayed_work(&sched->work_tdr) in drm_sched_cleanup_jobs - crash log attached in crash.log. This I think confirms my theory i descri
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Well then we should probably update the documentation. Take a look at the implementation, there is no compiler or SMP barrier at all. Christian. Am 18.11.19 um 18:01 schrieb Andrey Grodzovsky: The documentation states it can be used safely with concurrent list_del_init so I assume it's true - but I think my even bigger mistake is that without locking i just do list_first_entry right after list_empty_careful and by this can grab pointer to the same job as concurrent drm_sched_job_timedout->list_first_entry_or_null - so yes I see now i have to use locking as you advised there and then i don't need the list_empty_careful. Andrey On 11/18/19 11:44 AM, Christian König wrote: list_empty_careful() should only be used for optimizing cases, but never if you need to rely on the result. The problem is that the function doesn't has any memory barriers whatsoever, it just checks if the next and prev pointer are both empty instead of just the next pointer. Christian. Am 18.11.19 um 17:23 schrieb Andrey Grodzovsky: Can you explain why ? As I see it - list_empty_careful is specifically designed for the case where the only other concurrent operation in progress is list_del_init (https://www.kernel.org/doc/htmldocs/kernel-api/API-list-empty-careful.html) - which is exactly what happens in this patch, no other list altering operation can take place concurrently - so it looks safe to use for me. Andrey On 11/18/19 11:16 AM, Christian König wrote: Hi Andrey, the only thing which doesn't looks so good is the switch to list_empty_careful in drm_sched_cleanup_jobs. We either take the lock here or we don't, but please not that extra checking. Christian. Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky: Thanks Emily. Christan - ping for review. Andrey On 11/14/19 11:39 PM, Deng, Emily wrote: Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Friday, November 15, 2019 6:14 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Attached. Emily - can you give it a try ? Andrey On 11/14/19 3:12 AM, Christian König wrote: What about instead of peeking at the job to actually remove it from ring_mirror_list right there, Also an interesting idea. We would need to protect the mirror list with a lock again, but that should be the lesser evil. Maybe prototype that and see if it works or not. Regards, Christian. Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: On 11/13/19 9:20 AM, Christian König wrote: Another more fundamental question: Could we get rid of the timeout job at all? There are other stuff there besides picking the first unfinished job which is common for all the drivers - such as freeing guilty signaled job and rearming the timeout work timer. I mean we used to give this as parameter to the scheduler callback because we had the timeout worker in the job, but that is no longer the case. E.g. in drm_sched_job_timedout() we do the following: job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); Why don't we just remove that here and only get the first job after we have stopped the scheduler? Should be ok since we have the extra check for __kthread_should_park in drm_sched_cleanup_jobs which will protect us in this case from a wakeup of sched thread and execution of in drm_sched_cleanup_jobs after we already parked it. The problem here is we need the drm_sched_job to access the private data for each client driver (see amdgpu_job_timedout for example). What about instead of peeking at the job to actually remove it from ring_mirror_list right there, go ahead with it through the reset routine, if it's signaled in the meanwhile that great - release it, otherwise put it back into ring_mirror_list in drm_sched_resubmit_jobs. Andrey Regards, Christian. Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: This why I asked for a trace with timer enabled, but since there is a finite number of places we touch the timer Emily can just put prints there. Also, I wonder if this temp fix helps her with the issue or not. Andrey On 11/13/19 2:36 AM, Christian König wrote: The question is where do we rearm the timer for this problem to occur? Regards, Christian. Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: I was able to reproduce the crash by using the attached simulate_crash.patch - waiting on guilty job to signal in reset work and artificially rearming the timeout timer just before the check for !cancel_delayed_work(&sched->work_tdr) in drm_sched_cleanup_jobs - crash log attached in crash.log. This I think confirms my theory i described earlier in this thread. basic_fix.patch handles this by testing whether another
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
The documentation states it can be used safely with concurrent list_del_init so I assume it's true - but I think my even bigger mistake is that without locking i just do list_first_entry right after list_empty_careful and by this can grab pointer to the same job as concurrent drm_sched_job_timedout->list_first_entry_or_null - so yes I see now i have to use locking as you advised there and then i don't need the list_empty_careful. Andrey On 11/18/19 11:44 AM, Christian König wrote: list_empty_careful() should only be used for optimizing cases, but never if you need to rely on the result. The problem is that the function doesn't has any memory barriers whatsoever, it just checks if the next and prev pointer are both empty instead of just the next pointer. Christian. Am 18.11.19 um 17:23 schrieb Andrey Grodzovsky: Can you explain why ? As I see it - list_empty_careful is specifically designed for the case where the only other concurrent operation in progress is list_del_init (https://www.kernel.org/doc/htmldocs/kernel-api/API-list-empty-careful.html) - which is exactly what happens in this patch, no other list altering operation can take place concurrently - so it looks safe to use for me. Andrey On 11/18/19 11:16 AM, Christian König wrote: Hi Andrey, the only thing which doesn't looks so good is the switch to list_empty_careful in drm_sched_cleanup_jobs. We either take the lock here or we don't, but please not that extra checking. Christian. Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky: Thanks Emily. Christan - ping for review. Andrey On 11/14/19 11:39 PM, Deng, Emily wrote: Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Friday, November 15, 2019 6:14 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Attached. Emily - can you give it a try ? Andrey On 11/14/19 3:12 AM, Christian König wrote: What about instead of peeking at the job to actually remove it from ring_mirror_list right there, Also an interesting idea. We would need to protect the mirror list with a lock again, but that should be the lesser evil. Maybe prototype that and see if it works or not. Regards, Christian. Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: On 11/13/19 9:20 AM, Christian König wrote: Another more fundamental question: Could we get rid of the timeout job at all? There are other stuff there besides picking the first unfinished job which is common for all the drivers - such as freeing guilty signaled job and rearming the timeout work timer. I mean we used to give this as parameter to the scheduler callback because we had the timeout worker in the job, but that is no longer the case. E.g. in drm_sched_job_timedout() we do the following: job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); Why don't we just remove that here and only get the first job after we have stopped the scheduler? Should be ok since we have the extra check for __kthread_should_park in drm_sched_cleanup_jobs which will protect us in this case from a wakeup of sched thread and execution of in drm_sched_cleanup_jobs after we already parked it. The problem here is we need the drm_sched_job to access the private data for each client driver (see amdgpu_job_timedout for example). What about instead of peeking at the job to actually remove it from ring_mirror_list right there, go ahead with it through the reset routine, if it's signaled in the meanwhile that great - release it, otherwise put it back into ring_mirror_list in drm_sched_resubmit_jobs. Andrey Regards, Christian. Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: This why I asked for a trace with timer enabled, but since there is a finite number of places we touch the timer Emily can just put prints there. Also, I wonder if this temp fix helps her with the issue or not. Andrey On 11/13/19 2:36 AM, Christian König wrote: The question is where do we rearm the timer for this problem to occur? Regards, Christian. Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: I was able to reproduce the crash by using the attached simulate_crash.patch - waiting on guilty job to signal in reset work and artificially rearming the timeout timer just before the check for !cancel_delayed_work(&sched->work_tdr) in drm_sched_cleanup_jobs - crash log attached in crash.log. This I think confirms my theory i described earlier in this thread. basic_fix.patch handles this by testing whether another timer already armed ob this scheduler or is there a timeout work in execution right now (see documentation for work_busy) - obviously this is not a full solution as this will not protect from race
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
list_empty_careful() should only be used for optimizing cases, but never if you need to rely on the result. The problem is that the function doesn't has any memory barriers whatsoever, it just checks if the next and prev pointer are both empty instead of just the next pointer. Christian. Am 18.11.19 um 17:23 schrieb Andrey Grodzovsky: Can you explain why ? As I see it - list_empty_careful is specifically designed for the case where the only other concurrent operation in progress is list_del_init (https://www.kernel.org/doc/htmldocs/kernel-api/API-list-empty-careful.html) - which is exactly what happens in this patch, no other list altering operation can take place concurrently - so it looks safe to use for me. Andrey On 11/18/19 11:16 AM, Christian König wrote: Hi Andrey, the only thing which doesn't looks so good is the switch to list_empty_careful in drm_sched_cleanup_jobs. We either take the lock here or we don't, but please not that extra checking. Christian. Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky: Thanks Emily. Christan - ping for review. Andrey On 11/14/19 11:39 PM, Deng, Emily wrote: Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Friday, November 15, 2019 6:14 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Attached. Emily - can you give it a try ? Andrey On 11/14/19 3:12 AM, Christian König wrote: What about instead of peeking at the job to actually remove it from ring_mirror_list right there, Also an interesting idea. We would need to protect the mirror list with a lock again, but that should be the lesser evil. Maybe prototype that and see if it works or not. Regards, Christian. Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: On 11/13/19 9:20 AM, Christian König wrote: Another more fundamental question: Could we get rid of the timeout job at all? There are other stuff there besides picking the first unfinished job which is common for all the drivers - such as freeing guilty signaled job and rearming the timeout work timer. I mean we used to give this as parameter to the scheduler callback because we had the timeout worker in the job, but that is no longer the case. E.g. in drm_sched_job_timedout() we do the following: job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); Why don't we just remove that here and only get the first job after we have stopped the scheduler? Should be ok since we have the extra check for __kthread_should_park in drm_sched_cleanup_jobs which will protect us in this case from a wakeup of sched thread and execution of in drm_sched_cleanup_jobs after we already parked it. The problem here is we need the drm_sched_job to access the private data for each client driver (see amdgpu_job_timedout for example). What about instead of peeking at the job to actually remove it from ring_mirror_list right there, go ahead with it through the reset routine, if it's signaled in the meanwhile that great - release it, otherwise put it back into ring_mirror_list in drm_sched_resubmit_jobs. Andrey Regards, Christian. Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: This why I asked for a trace with timer enabled, but since there is a finite number of places we touch the timer Emily can just put prints there. Also, I wonder if this temp fix helps her with the issue or not. Andrey On 11/13/19 2:36 AM, Christian König wrote: The question is where do we rearm the timer for this problem to occur? Regards, Christian. Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: I was able to reproduce the crash by using the attached simulate_crash.patch - waiting on guilty job to signal in reset work and artificially rearming the timeout timer just before the check for !cancel_delayed_work(&sched->work_tdr) in drm_sched_cleanup_jobs - crash log attached in crash.log. This I think confirms my theory i described earlier in this thread. basic_fix.patch handles this by testing whether another timer already armed ob this scheduler or is there a timeout work in execution right now (see documentation for work_busy) - obviously this is not a full solution as this will not protect from races if for example there is immediate work scheduling such as in drm_sched_fault - so we probably need to account for this by making drm_sched_cleanup_jobs (at least in the part where it iterates ring mirror list and frees jobs) and GPU reset really mutually exclusive and not like now. Andrey On 11/11/19 4:11 PM, Christian König wrote: Hi Emily, you need to print which scheduler instance is freeing the jobs and which one is triggering the reset. The TID and PID is completely meaningless here since we are c
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Can you explain why ? As I see it - list_empty_careful is specifically designed for the case where the only other concurrent operation in progress is list_del_init (https://www.kernel.org/doc/htmldocs/kernel-api/API-list-empty-careful.html) - which is exactly what happens in this patch, no other list altering operation can take place concurrently - so it looks safe to use for me. Andrey On 11/18/19 11:16 AM, Christian König wrote: Hi Andrey, the only thing which doesn't looks so good is the switch to list_empty_careful in drm_sched_cleanup_jobs. We either take the lock here or we don't, but please not that extra checking. Christian. Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky: Thanks Emily. Christan - ping for review. Andrey On 11/14/19 11:39 PM, Deng, Emily wrote: Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Friday, November 15, 2019 6:14 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Attached. Emily - can you give it a try ? Andrey On 11/14/19 3:12 AM, Christian König wrote: What about instead of peeking at the job to actually remove it from ring_mirror_list right there, Also an interesting idea. We would need to protect the mirror list with a lock again, but that should be the lesser evil. Maybe prototype that and see if it works or not. Regards, Christian. Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: On 11/13/19 9:20 AM, Christian König wrote: Another more fundamental question: Could we get rid of the timeout job at all? There are other stuff there besides picking the first unfinished job which is common for all the drivers - such as freeing guilty signaled job and rearming the timeout work timer. I mean we used to give this as parameter to the scheduler callback because we had the timeout worker in the job, but that is no longer the case. E.g. in drm_sched_job_timedout() we do the following: job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); Why don't we just remove that here and only get the first job after we have stopped the scheduler? Should be ok since we have the extra check for __kthread_should_park in drm_sched_cleanup_jobs which will protect us in this case from a wakeup of sched thread and execution of in drm_sched_cleanup_jobs after we already parked it. The problem here is we need the drm_sched_job to access the private data for each client driver (see amdgpu_job_timedout for example). What about instead of peeking at the job to actually remove it from ring_mirror_list right there, go ahead with it through the reset routine, if it's signaled in the meanwhile that great - release it, otherwise put it back into ring_mirror_list in drm_sched_resubmit_jobs. Andrey Regards, Christian. Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: This why I asked for a trace with timer enabled, but since there is a finite number of places we touch the timer Emily can just put prints there. Also, I wonder if this temp fix helps her with the issue or not. Andrey On 11/13/19 2:36 AM, Christian König wrote: The question is where do we rearm the timer for this problem to occur? Regards, Christian. Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: I was able to reproduce the crash by using the attached simulate_crash.patch - waiting on guilty job to signal in reset work and artificially rearming the timeout timer just before the check for !cancel_delayed_work(&sched->work_tdr) in drm_sched_cleanup_jobs - crash log attached in crash.log. This I think confirms my theory i described earlier in this thread. basic_fix.patch handles this by testing whether another timer already armed ob this scheduler or is there a timeout work in execution right now (see documentation for work_busy) - obviously this is not a full solution as this will not protect from races if for example there is immediate work scheduling such as in drm_sched_fault - so we probably need to account for this by making drm_sched_cleanup_jobs (at least in the part where it iterates ring mirror list and frees jobs) and GPU reset really mutually exclusive and not like now. Andrey On 11/11/19 4:11 PM, Christian König wrote: Hi Emily, you need to print which scheduler instance is freeing the jobs and which one is triggering the reset. The TID and PID is completely meaningless here since we are called from different worker threads and the TID/PID can change on each call. Apart from that I will look into this a bit deeper when I have time. Regards, Christian. Am 12.11.19 um 07:02 schrieb Deng, Emily: Hi Christian, I add the follow print in function drm_sched_cleanup_jobs. From the log it shows that only use cancel_delayed_work could not avoi
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Andrey,
the only thing which doesn't looks so good is the switch to
list_empty_careful in drm_sched_cleanup_jobs.
We either take the lock here or we don't, but please not that extra
checking.
Christian.
Am 18.11.19 um 15:07 schrieb Andrey Grodzovsky:
Thanks Emily.
Christan - ping for review.
Andrey
On 11/14/19 11:39 PM, Deng, Emily wrote:
Hi Andrey,
Currently, I am busying with another issue, maybe will try next
week.
Best wishes
Emily Deng
-Original Message-
From: Grodzovsky, Andrey
Sent: Friday, November 15, 2019 6:14 AM
To: Koenig, Christian ; Deng, Emily
; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Attached.
Emily - can you give it a try ?
Andrey
On 11/14/19 3:12 AM, Christian König wrote:
What about instead of peeking at the job to actually remove it from
ring_mirror_list right there,
Also an interesting idea. We would need to protect the mirror list
with a lock again, but that should be the lesser evil.
Maybe prototype that and see if it works or not.
Regards,
Christian.
Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky:
On 11/13/19 9:20 AM, Christian König wrote:
Another more fundamental question: Could we get rid of the timeout
job at all?
There are other stuff there besides picking the first unfinished job
which is common for all the drivers - such as freeing guilty signaled
job and rearming the timeout work timer.
I mean we used to give this as parameter to the scheduler callback
because we had the timeout worker in the job, but that is no longer
the case.
E.g. in drm_sched_job_timedout() we do the following:
job = list_first_entry_or_null(&sched->ring_mirror_list,
struct drm_sched_job,
node);
Why don't we just remove that here and only get the first job after
we have stopped the scheduler?
Should be ok since we have the extra check for __kthread_should_park
in drm_sched_cleanup_jobs which will protect us in this case from a
wakeup of sched thread and execution of in drm_sched_cleanup_jobs
after we already parked it. The problem here is we need the
drm_sched_job to access the private data for each client driver (see
amdgpu_job_timedout for example). What about instead of peeking at
the job to actually remove it from ring_mirror_list right there, go
ahead with it through the reset routine, if it's signaled in the
meanwhile that great - release it, otherwise put it back into
ring_mirror_list in drm_sched_resubmit_jobs.
Andrey
Regards,
Christian.
Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky:
This why I asked for a trace with timer enabled, but since there is
a finite number of places we touch the timer Emily can just put
prints there. Also, I wonder if this temp fix helps her with the
issue or not.
Andrey
On 11/13/19 2:36 AM, Christian König wrote:
The question is where do we rearm the timer for this problem to
occur?
Regards,
Christian.
Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky:
I was able to reproduce the crash by using the attached
simulate_crash.patch - waiting on guilty job to signal in reset
work and artificially rearming the timeout timer just before the
check for !cancel_delayed_work(&sched->work_tdr) in
drm_sched_cleanup_jobs - crash log attached in crash.log. This I
think confirms my theory i described earlier in this thread.
basic_fix.patch handles this by testing whether another timer
already armed ob this scheduler or is there a timeout work in
execution right now (see documentation for work_busy) - obviously
this is not a full solution as this will not protect from races
if for example there is immediate work scheduling such as in
drm_sched_fault - so we probably need to account for this by
making drm_sched_cleanup_jobs (at least in the part where it
iterates ring mirror list and frees jobs) and GPU reset really
mutually exclusive and not like now.
Andrey
On 11/11/19 4:11 PM, Christian König wrote:
Hi Emily,
you need to print which scheduler instance is freeing the jobs
and which one is triggering the reset. The TID and PID is
completely meaningless here since we are called from different
worker threads and the TID/PID can change on each call.
Apart from that I will look into this a bit deeper when I have
time.
Regards,
Christian.
Am 12.11.19 um 07:02 schrieb Deng, Emily:
Hi Christian,
I add the follow print in function drm_sched_cleanup_jobs.
From the log it shows that only use cancel_delayed_work could
not avoid to free job when the sched is in reset. But don’t
know exactly where it is wrong about the driver. Do you have
any suggestion about this?
+ printk("Emily:drm_sched_cleanup_jobs:begin,tid:%lu,
pid:%lu\n", current->tgid, current->pid);
/*
* Don't destroy jobs while the timeout worker is
running OR thread
* is being parked and hence assumed to not touch
ring_mirror_list
*
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Thanks Emily.
Christan - ping for review.
Andrey
On 11/14/19 11:39 PM, Deng, Emily wrote:
Hi Andrey,
Currently, I am busying with another issue, maybe will try next week.
Best wishes
Emily Deng
-Original Message-
From: Grodzovsky, Andrey
Sent: Friday, November 15, 2019 6:14 AM
To: Koenig, Christian ; Deng, Emily
; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Attached.
Emily - can you give it a try ?
Andrey
On 11/14/19 3:12 AM, Christian König wrote:
What about instead of peeking at the job to actually remove it from
ring_mirror_list right there,
Also an interesting idea. We would need to protect the mirror list
with a lock again, but that should be the lesser evil.
Maybe prototype that and see if it works or not.
Regards,
Christian.
Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky:
On 11/13/19 9:20 AM, Christian König wrote:
Another more fundamental question: Could we get rid of the timeout
job at all?
There are other stuff there besides picking the first unfinished job
which is common for all the drivers - such as freeing guilty signaled
job and rearming the timeout work timer.
I mean we used to give this as parameter to the scheduler callback
because we had the timeout worker in the job, but that is no longer
the case.
E.g. in drm_sched_job_timedout() we do the following:
job = list_first_entry_or_null(&sched->ring_mirror_list,
struct drm_sched_job, node);
Why don't we just remove that here and only get the first job after
we have stopped the scheduler?
Should be ok since we have the extra check for __kthread_should_park
in drm_sched_cleanup_jobs which will protect us in this case from a
wakeup of sched thread and execution of in drm_sched_cleanup_jobs
after we already parked it. The problem here is we need the
drm_sched_job to access the private data for each client driver (see
amdgpu_job_timedout for example). What about instead of peeking at
the job to actually remove it from ring_mirror_list right there, go
ahead with it through the reset routine, if it's signaled in the
meanwhile that great - release it, otherwise put it back into
ring_mirror_list in drm_sched_resubmit_jobs.
Andrey
Regards,
Christian.
Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky:
This why I asked for a trace with timer enabled, but since there is
a finite number of places we touch the timer Emily can just put
prints there. Also, I wonder if this temp fix helps her with the
issue or not.
Andrey
On 11/13/19 2:36 AM, Christian König wrote:
The question is where do we rearm the timer for this problem to
occur?
Regards,
Christian.
Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky:
I was able to reproduce the crash by using the attached
simulate_crash.patch - waiting on guilty job to signal in reset
work and artificially rearming the timeout timer just before the
check for !cancel_delayed_work(&sched->work_tdr) in
drm_sched_cleanup_jobs - crash log attached in crash.log. This I
think confirms my theory i described earlier in this thread.
basic_fix.patch handles this by testing whether another timer
already armed ob this scheduler or is there a timeout work in
execution right now (see documentation for work_busy) - obviously
this is not a full solution as this will not protect from races
if for example there is immediate work scheduling such as in
drm_sched_fault - so we probably need to account for this by
making drm_sched_cleanup_jobs (at least in the part where it
iterates ring mirror list and frees jobs) and GPU reset really
mutually exclusive and not like now.
Andrey
On 11/11/19 4:11 PM, Christian König wrote:
Hi Emily,
you need to print which scheduler instance is freeing the jobs
and which one is triggering the reset. The TID and PID is
completely meaningless here since we are called from different
worker threads and the TID/PID can change on each call.
Apart from that I will look into this a bit deeper when I have
time.
Regards,
Christian.
Am 12.11.19 um 07:02 schrieb Deng, Emily:
Hi Christian,
I add the follow print in function drm_sched_cleanup_jobs.
From the log it shows that only use cancel_delayed_work could
not avoid to free job when the sched is in reset. But don’t
know exactly where it is wrong about the driver. Do you have
any suggestion about this?
+ printk("Emily:drm_sched_cleanup_jobs:begin,tid:%lu,
pid:%lu\n", current->tgid, current->pid);
/*
* Don't destroy jobs while the timeout worker is
running OR thread
* is being parked and hence assumed to not touch
ring_mirror_list
*/
if ((sched->timeout != MAX_SCHEDULE_TIMEOUT &&
!cancel_delayed_work(&sched->work_tdr)))
return;
+ printk("Emily:drm_sched_cleanup_jobs,tid:%lu, pid:%lu\n",
current->tgid, current->pid);
Best wishes
Emily D
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Andrey, Currently, I am busying with another issue, maybe will try next week. Best wishes Emily Deng >-Original Message- >From: Grodzovsky, Andrey >Sent: Friday, November 15, 2019 6:14 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Attached. > >Emily - can you give it a try ? > >Andrey > >On 11/14/19 3:12 AM, Christian König wrote: >>> What about instead of peeking at the job to actually remove it from >>> ring_mirror_list right there, >> Also an interesting idea. We would need to protect the mirror list >> with a lock again, but that should be the lesser evil. >> >> Maybe prototype that and see if it works or not. >> >> Regards, >> Christian. >> >> Am 13.11.19 um 17:00 schrieb Andrey Grodzovsky: >>> >>> >>> On 11/13/19 9:20 AM, Christian König wrote: >>>> Another more fundamental question: Could we get rid of the timeout >>>> job at all? >>> >>> >>> There are other stuff there besides picking the first unfinished job >>> which is common for all the drivers - such as freeing guilty signaled >>> job and rearming the timeout work timer. >>> >>> >>>> >>>> I mean we used to give this as parameter to the scheduler callback >>>> because we had the timeout worker in the job, but that is no longer >>>> the case. >>>> >>>> E.g. in drm_sched_job_timedout() we do the following: >>>>> job = list_first_entry_or_null(&sched->ring_mirror_list, >>>>> struct drm_sched_job, node); >>>> >>>> Why don't we just remove that here and only get the first job after >>>> we have stopped the scheduler? >>> >>> >>> Should be ok since we have the extra check for __kthread_should_park >>> in drm_sched_cleanup_jobs which will protect us in this case from a >>> wakeup of sched thread and execution of in drm_sched_cleanup_jobs >>> after we already parked it. The problem here is we need the >>> drm_sched_job to access the private data for each client driver (see >>> amdgpu_job_timedout for example). What about instead of peeking at >>> the job to actually remove it from ring_mirror_list right there, go >>> ahead with it through the reset routine, if it's signaled in the >>> meanwhile that great - release it, otherwise put it back into >>> ring_mirror_list in drm_sched_resubmit_jobs. >>> >>> Andrey >>> >>> >>>> >>>> Regards, >>>> Christian. >>>> >>>> Am 13.11.19 um 15:12 schrieb Andrey Grodzovsky: >>>>> >>>>> This why I asked for a trace with timer enabled, but since there is >>>>> a finite number of places we touch the timer Emily can just put >>>>> prints there. Also, I wonder if this temp fix helps her with the >>>>> issue or not. >>>>> >>>>> Andrey >>>>> >>>>> On 11/13/19 2:36 AM, Christian König wrote: >>>>>> The question is where do we rearm the timer for this problem to >>>>>> occur? >>>>>> >>>>>> Regards, >>>>>> Christian. >>>>>> >>>>>> Am 12.11.19 um 20:21 schrieb Andrey Grodzovsky: >>>>>>> >>>>>>> I was able to reproduce the crash by using the attached >>>>>>> simulate_crash.patch - waiting on guilty job to signal in reset >>>>>>> work and artificially rearming the timeout timer just before the >>>>>>> check for !cancel_delayed_work(&sched->work_tdr) in >>>>>>> drm_sched_cleanup_jobs - crash log attached in crash.log. This I >>>>>>> think confirms my theory i described earlier in this thread. >>>>>>> >>>>>>> basic_fix.patch handles this by testing whether another timer >>>>>>> already armed ob this scheduler or is there a timeout work in >>>>>>> execution right now (see documentation for work_busy) - obviously >>>>>>> this is not a full solution as this will not protect from races >>>>>>> if for example there is immediate work scheduling such as in >>>>>>> drm_sched_fault - so we probably need to account for this by
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
62 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.222954] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=78585, emitted seq=78587 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.224275] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:fe75ab36,tid=15603, pid=15603 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225413] amdgpu :00:08.0: GPU reset begin! Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225417] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225428] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:fe75ab36, tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >____________________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >> !cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>&
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
kernel: [11380.695107] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.222954] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=78585, emitted seq=78587 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.224275] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:fe75ab36,tid=15603, pid=15603 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225413] amdgpu :00:08.0: GPU reset begin! Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225417] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225428] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:fe75ab36, tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >____________________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >> !cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.222954] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=78585, emitted seq=78587 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.224275] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:fe75ab36,tid=15603, pid=15603 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225413] amdgpu :00:08.0: GPU reset begin! Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225417] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225428] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:fe75ab36, tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >____________________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Chris
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
drop-August-2018-rc2-gpu0-vf02 kernel: [11381.224275] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:fe75ab36,tid=15603, pid=15603 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225413] amdgpu :00:08.0: GPU reset begin! Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225417] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225428] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:fe75ab36, tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >____________________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdg
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
v 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >________________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Autho
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
m_sched_cleanup_jobs,tid:2262, pid:2262 Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262 >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Author: Christian König >>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>> >>>>>> drm/scheduler: rework job destruction >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> Please refer to follow log, when it enter to >>>>>> amdgpu_device_gpu_recover >>>>> function, the bad job 5086879e is freeing in function >>>>> am
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
ssage- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 11:28 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Author: Christian König >>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>> >>>>>> drm/scheduler: rework job destruction >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> Please refer to follow log, when it enter to >>>>>> amdgpu_device_gpu_recover >>>>> function, the bad job 5086879e is freeing in function >>>>> amdgpu_job_free_cb at the same time, because of the hardware fence >>> signal. >>>>> But amdgpu_device_gpu_recover goes faster, at this case, the >>>>> s_fence is already freed, but job is not freed in time. Then this issue >occurs. &
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Thinking more about this claim - we assume here that if cancel_delayed_work >returned true it guarantees that timeout work is not running but, it merely >means there was a pending timeout work which was removed from the >workqueue before it's timer elapsed and so it didn't have a chance to be >dequeued and executed, it doesn't cover already executing work. So there is a >possibility where while timeout work started executing another timeout work >already got enqueued (maybe through earlier cleanup jobs or through >drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs >cancel_delayed_work(&sched->work_tdr) will return true even while there is a >timeout job in progress. >Unfortunately we cannot change cancel_delayed_work to >cancel_delayed_work_sync to flush the timeout work as timeout work itself >waits for schedule thread to be parked again when calling park_thread. > >Andrey > >________ >From: amd-gfx on behalf of >Koenig, Christian >Sent: 08 November 2019 05:35:18 >To: Deng, Emily; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>!cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Author: Christian König >>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>> >>>>>> drm/scheduler: rework job destruction >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> Please refer to follow log, when it enter to >>>>>> amdgpu_device_gpu_recover >>>>> function, the bad job 5086879e is freeing in function >>>>> amdgpu_job_free_cb at the same time, because of the hardware fence >>> signal. >>>>> But amdgpu_device_gpu_recover goes faster, at this case, the >>>>> s_fence is already freed, but job is not freed in time. Then this issue >occurs. >>>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >>> sdma0 >>>>>> timeo
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Emily,
you need to print which scheduler instance is freeing the jobs and which
one is triggering the reset. The TID and PID is completely meaningless
here since we are called from different worker threads and the TID/PID
can change on each call.
Apart from that I will look into this a bit deeper when I have time.
Regards,
Christian.
Am 12.11.19 um 07:02 schrieb Deng, Emily:
Hi Christian,
I add the follow print in function drm_sched_cleanup_jobs. From
the log it shows that only use cancel_delayed_work could not avoid to
free job when the sched is in reset. But don’t know exactly where it
is wrong about the driver. Do you have any suggestion about this?
+ printk("Emily:drm_sched_cleanup_jobs:begin,tid:%lu, pid:%lu\n",
current->tgid, current->pid);
/*
* Don't destroy jobs while the timeout worker is running OR
thread
* is being parked and hence assumed to not touch ring_mirror_list
*/
if ((sched->timeout != MAX_SCHEDULE_TIMEOUT &&
!cancel_delayed_work(&sched->work_tdr)))
return;
+ printk("Emily:drm_sched_cleanup_jobs,tid:%lu, pid:%lu\n",
current->tgid, current->pid);
Best wishes
Emily Deng
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11380.695091] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11380.695104] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11380.695105] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11380.695107] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11380.695107] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.222954] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0
timeout, signaled seq=78585, emitted seq=78587
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.224275] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process
information: process pid 0 thread pid 0,
s_job:fe75ab36,tid=15603, pid=15603
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225413] amdgpu :00:08.0: GPU reset begin!
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225417] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225425] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225425] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225428] Emily:amdgpu_job_free_cb,Process information: process
pid 0 thread pid 0, s_job:fe75ab36, tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225429] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225430] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225473] Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225486] Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225489] Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel:
[11381.225494] Emily:amdgpu_job_free_cb,Process information: process
pid 0 thread pid 0, s_job:f086ec84, tid:2262, pid:2262
>-Original Message-
>From: Grodzovsky, Andrey
>Sent: Tuesday, November 12, 2019 11:28 AM
>To: Koenig, Christian ; Deng, Emily
>; amd-gfx@lists.freedesktop.org
>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>Thinking more about this claim - we assume here that if cancel_delayed_work
>returned true it guarantees that timeout work is not running but, it merely
>means there was a pending timeout work which was removed from the
>workqueue before it's timer elapsed and so it didn't have a chance to be
>dequeued and executed, it doesn't cover already executing work. So there is a
>possibility where while timeout work started executing another timeout work
>already got enqueued (maybe through earlier cleanup jobs or through
>drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs
>cancel_delayed_work(&sched->work_tdr) will return true even while there is a
>timeout job in progress.
>Unfortunately we cannot change cancel_delayed_work to
>cancel_delayed_work_sync to flush the timeout work as timeout work itsel
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Christian,
I add the follow print in function drm_sched_cleanup_jobs. From the log it
shows that only use cancel_delayed_work could not avoid to free job when the
sched is in reset. But don't know exactly where it is wrong about the driver.
Do you have any suggestion about this?
+ printk("Emily:drm_sched_cleanup_jobs:begin,tid:%lu, pid:%lu\n",
current->tgid, current->pid);
/*
* Don't destroy jobs while the timeout worker is running OR thread
* is being parked and hence assumed to not touch ring_mirror_list
*/
if ((sched->timeout != MAX_SCHEDULE_TIMEOUT &&
!cancel_delayed_work(&sched->work_tdr)))
return;
+ printk("Emily:drm_sched_cleanup_jobs,tid:%lu, pid:%lu\n",
current->tgid, current->pid);
Best wishes
Emily Deng
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695091]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695104]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695105]
Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11380.695107]
Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.222954]
[drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled
seq=78585, emitted seq=78587
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.224275]
[drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0
thread pid 0, s_job:fe75ab36,tid=15603, pid=15603
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225413]
amdgpu :00:08.0: GPU reset begin!
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225417]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225425]
Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225428]
Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0,
s_job:fe75ab36, tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225429]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225430]
Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225473]
Emily:drm_sched_cleanup_jobs:begin,tid:2253, pid:2253
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225486]
Emily:drm_sched_cleanup_jobs:begin,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225489]
Emily:drm_sched_cleanup_jobs,tid:2262, pid:2262
Nov 12 12:58:20 ubuntu-drop-August-2018-rc2-gpu0-vf02 kernel: [11381.225494]
Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0,
s_job:f086ec84, tid:2262, pid:2262
>-Original Message-
>From: Grodzovsky, Andrey
>Sent: Tuesday, November 12, 2019 11:28 AM
>To: Koenig, Christian ; Deng, Emily
>; amd-gfx@lists.freedesktop.org
>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>Thinking more about this claim - we assume here that if cancel_delayed_work
>returned true it guarantees that timeout work is not running but, it merely
>means there was a pending timeout work which was removed from the
>workqueue before it's timer elapsed and so it didn't have a chance to be
>dequeued and executed, it doesn't cover already executing work. So there is a
>possibility where while timeout work started executing another timeout work
>already got enqueued (maybe through earlier cleanup jobs or through
>drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs
>cancel_delayed_work(&sched->work_tdr) will return true even while there is a
>timeout job in progress.
>Unfortunately we cannot change cancel_delayed_work to
>cancel_delayed_work_sync to flush the timeout work as timeout work itself
>waits for schedule thread to be parked again when calling park_thread.
>
>Andrey
>
>____
>From: amd-gfx on behalf of
>Koenig, Christian
>Sent: 08 November 2019 05:35:18
>To: Deng, Emily; amd-gfx@lists.freedesktop.org
>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>Hi
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Andrey, On my side, it doesn't need to a specific scenario, I only run the quark with slow job. Then sometimes, it will have fake hang and hardware fence will back. For this case, it will randomly occur the NULL pointer issue in amdgpu_device_gpu_recover. >-Original Message- >From: Grodzovsky, Andrey >Sent: Tuesday, November 12, 2019 5:35 AM >To: Deng, Emily ; Koenig, Christian >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Emily - is there a particular scenario to reproduce this ? I am trying with >libdrm >deadlock test and artificially delaying the GPU reset logic until after the >guilty >job is signaling but indeed nothing bad happens as drm_sched_cleanup_jobs >returns early because there is a reset in progress and so the bad job is not >getting released while GPU reset is running. > >Can you provide event tracing for timer, dma_fence and gpu_scheduler for >when the problem happens ? > >Andrey > >On 11/11/19 4:05 AM, Deng, Emily wrote: >> Hi Christian and Andrey, >> The issue I encountered is the bad job is freeing after entering to the >amdgpu_device_gpu_recover. Don't know why, as per Christian said, it will >call cancel_delayed_work in drm_sched_cleanup_jobs. >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: amd-gfx On Behalf Of >>> Deng, Emily >>> Sent: Monday, November 11, 2019 3:19 PM >>> To: Grodzovsky, Andrey ; Koenig, >Christian >>> ; amd-gfx@lists.freedesktop.org >>> Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Andrey, >>> I don’t think your patch will help for this. As it will may call >>> kthread_should_park in drm_sched_cleanup_jobs first, and then call >>> kcl_kthread_park. And then it still has a race between the 2 threads. >>> >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: Grodzovsky, Andrey >>>> Sent: Saturday, November 9, 2019 3:01 AM >>>> To: Koenig, Christian ; Deng, Emily >>>> ; amd-gfx@lists.freedesktop.org >>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>> >>>> On 11/8/19 5:35 AM, Koenig, Christian wrote: >>>>> Hi Emily, >>>>> >>>>> exactly that can't happen. See here: >>>>> >>>>>> /* Don't destroy jobs while the timeout worker is >>>>>> running */ >>>>>> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>>>>> !cancel_delayed_work(&sched->work_tdr)) >>>>>> return NULL; >>>>> We never free jobs while the timeout working is running to prevent >>>>> exactly that issue. >>>> >>>> I don't think this protects us if drm_sched_cleanup_jobs is called >>>> for scheduler which didn't experience a timeout, in >>>> amdgpu_device_gpu_recover we access >>>> sched->ring_mirror_list for all the schedulers on a device so this >>>> sched->condition >>>> above won't protect us. What in fact could help maybe is my recent >>>> patch >>>> 541c521 drm/sched: Avoid job cleanup if sched thread is parked. >>>> because we do park each of the scheduler threads during tdr job >>>> before trying to access >>>> sched->ring_mirror_list. >>>> >>>> Emily - did you see this problem with that patch in place ? I only >>>> pushed it yesterday. >>>> >>>> Andrey >>>> >>>> >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:32 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> The drm_sched_job_timedout-> amdgpu_job_timedout call >>>> amdgpu_device_gpu_recover. I mean the main scheduler free the jobs >>>> while in amdgpu_device_gpu_recover, and before calling >drm_sched_stop. >>>>>> Best wishes >>>>>> Emily Deng >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Koenig, Christian >>>>>>> Sent: Friday, November 8, 2019 6:26 PM >>>>>>> To: Deng, Emily ; amd- >>>
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Thinking more about this claim - we assume here that if cancel_delayed_work returned true it guarantees that timeout work is not running but, it merely means there was a pending timeout work which was removed from the workqueue before it's timer elapsed and so it didn't have a chance to be dequeued and executed, it doesn't cover already executing work. So there is a possibility where while timeout work started executing another timeout work already got enqueued (maybe through earlier cleanup jobs or through drm_sched_fault) and if at this point another drm_sched_cleanup_jobs runs cancel_delayed_work(&sched->work_tdr) will return true even while there is a timeout job in progress. Unfortunately we cannot change cancel_delayed_work to cancel_delayed_work_sync to flush the timeout work as timeout work itself waits for schedule thread to be parked again when calling park_thread. Andrey From: amd-gfx on behalf of Koenig, Christian Sent: 08 November 2019 05:35:18 To: Deng, Emily; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Emily, exactly that can't happen. See here: > /* Don't destroy jobs while the timeout worker is running */ > if (sched->timeout != MAX_SCHEDULE_TIMEOUT && > !cancel_delayed_work(&sched->work_tdr)) > return NULL; We never free jobs while the timeout working is running to prevent exactly that issue. Regards, Christian. Am 08.11.19 um 11:32 schrieb Deng, Emily: > Hi Christian, > The drm_sched_job_timedout-> amdgpu_job_timedout call > amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in > amdgpu_device_gpu_recover, and before calling drm_sched_stop. > > Best wishes > Emily Deng > > > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 6:26 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Hi Emily, >> >> well who is calling amdgpu_device_gpu_recover() in this case? >> >> When it's not the scheduler we shouldn't have a guilty job in the first >> place. >> >> Regards, >> Christian. >> >> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>> Hi Chrisitan, >>>No, I am with the new branch and also has the patch. Even it are >>> freed by >> main scheduler, how we could avoid main scheduler to free jobs while enter >> to function amdgpu_device_gpu_recover? >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: Koenig, Christian >>>> Sent: Friday, November 8, 2019 6:15 PM >>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>> Hi Emily, >>>> >>>> in this case you are on an old code branch. >>>> >>>> Jobs are freed now by the main scheduler thread and only if no >>>> timeout handler is running. >>>> >>>> See this patch here: >>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>> Author: Christian König >>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>> >>>>> drm/scheduler: rework job destruction >>>> Regards, >>>> Christian. >>>> >>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>> Hi Christian, >>>>> Please refer to follow log, when it enter to >>>>> amdgpu_device_gpu_recover >>>> function, the bad job 5086879e is freeing in function >>>> amdgpu_job_free_cb at the same time, because of the hardware fence >> signal. >>>> But amdgpu_device_gpu_recover goes faster, at this case, the s_fence >>>> is already freed, but job is not freed in time. Then this issue occurs. >>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >> sdma0 >>>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >>>> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] >>>> amdgpu >>>> :00:08.0: GPU reset begin! >>>>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >>>>> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>>>> Emily:amdgpu_job_
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Emily - is there a particular scenario to reproduce this ? I am trying with libdrm deadlock test and artificially delaying the GPU reset logic until after the guilty job is signaling but indeed nothing bad happens as drm_sched_cleanup_jobs returns early because there is a reset in progress and so the bad job is not getting released while GPU reset is running. Can you provide event tracing for timer, dma_fence and gpu_scheduler for when the problem happens ? Andrey On 11/11/19 4:05 AM, Deng, Emily wrote: Hi Christian and Andrey, The issue I encountered is the bad job is freeing after entering to the amdgpu_device_gpu_recover. Don't know why, as per Christian said, it will call cancel_delayed_work in drm_sched_cleanup_jobs. Best wishes Emily Deng -Original Message- From: amd-gfx On Behalf Of Deng, Emily Sent: Monday, November 11, 2019 3:19 PM To: Grodzovsky, Andrey ; Koenig, Christian ; amd-gfx@lists.freedesktop.org Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Andrey, I don’t think your patch will help for this. As it will may call kthread_should_park in drm_sched_cleanup_jobs first, and then call kcl_kthread_park. And then it still has a race between the 2 threads. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Saturday, November 9, 2019 3:01 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr On 11/8/19 5:35 AM, Koenig, Christian wrote: Hi Emily, exactly that can't happen. See here: /* Don't destroy jobs while the timeout worker is running */ if (sched->timeout != MAX_SCHEDULE_TIMEOUT && !cancel_delayed_work(&sched->work_tdr)) return NULL; We never free jobs while the timeout working is running to prevent exactly that issue. I don't think this protects us if drm_sched_cleanup_jobs is called for scheduler which didn't experience a timeout, in amdgpu_device_gpu_recover we access sched->ring_mirror_list for all the schedulers on a device so this sched->condition above won't protect us. What in fact could help maybe is my recent patch 541c521 drm/sched: Avoid job cleanup if sched thread is parked. because we do park each of the scheduler threads during tdr job before trying to access sched->ring_mirror_list. Emily - did you see this problem with that patch in place ? I only pushed it yesterday. Andrey Regards, Christian. Am 08.11.19 um 11:32 schrieb Deng, Emily: Hi Christian, The drm_sched_job_timedout-> amdgpu_job_timedout call amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in amdgpu_device_gpu_recover, and before calling drm_sched_stop. Best wishes Emily Deng -Original Message- From: Koenig, Christian Sent: Friday, November 8, 2019 6:26 PM To: Deng, Emily ; amd- g...@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Emily, well who is calling amdgpu_device_gpu_recover() in this case? When it's not the scheduler we shouldn't have a guilty job in the first place. Regards, Christian. Am 08.11.19 um 11:22 schrieb Deng, Emily: Hi Chrisitan, No, I am with the new branch and also has the patch. Even it are freed by main scheduler, how we could avoid main scheduler to free jobs while enter to function amdgpu_device_gpu_recover? Best wishes Emily Deng -Original Message- From: Koenig, Christian Sent: Friday, November 8, 2019 6:15 PM To: Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Emily, in this case you are on an old code branch. Jobs are freed now by the main scheduler thread and only if no timeout handler is running. See this patch here: commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e Author: Christian König Date: Thu Apr 18 11:00:21 2019 -0400 drm/scheduler: rework job destruction Regards, Christian. Am 08.11.19 um 11:11 schrieb Deng, Emily: Hi Christian, Please refer to follow log, when it enter to amdgpu_device_gpu_recover function, the bad job 5086879e is freeing in function amdgpu_job_free_cb at the same time, because of the hardware fence signal. But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is already freed, but job is not freed in time. Then this issue occurs. [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794163] amdgpu :00:08.0: GPU reset begin! [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794221] Emily:amdgpu_job_free_cb,Process inform
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Note that kthread_park waits for kthread->parked to be signaled before proceeding - so in the scenario you described it meas main thread is running (not parked and so kthread->parked is not signaled) and so kthread_park will not proceed until the sched thread finish current loop (including removing any signaled jobs from ring_mirror_list) and is back to wait_event_interruptible->drm_sched_blocked->kthread_parkme->complete(&self->parked) to park itself - so looks to me it should be OK. Andrey On 11/11/19 2:19 AM, Deng, Emily wrote: Hi Andrey, I don’t think your patch will help for this. As it will may call kthread_should_park in drm_sched_cleanup_jobs first, and then call kcl_kthread_park. And then it still has a race between the 2 threads. Best wishes Emily Deng -Original Message- From: Grodzovsky, Andrey Sent: Saturday, November 9, 2019 3:01 AM To: Koenig, Christian ; Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr On 11/8/19 5:35 AM, Koenig, Christian wrote: Hi Emily, exactly that can't happen. See here: /* Don't destroy jobs while the timeout worker is running */ if (sched->timeout != MAX_SCHEDULE_TIMEOUT && !cancel_delayed_work(&sched->work_tdr)) return NULL; We never free jobs while the timeout working is running to prevent exactly that issue. I don't think this protects us if drm_sched_cleanup_jobs is called for scheduler which didn't experience a timeout, in amdgpu_device_gpu_recover we access sched->ring_mirror_list for all the schedulers on a device so this condition above won't protect us. What in fact could help maybe is my recent patch 541c521 drm/sched: Avoid job cleanup if sched thread is parked. because we do park each of the scheduler threads during tdr job before trying to access sched->ring_mirror_list. Emily - did you see this problem with that patch in place ? I only pushed it yesterday. Andrey Regards, Christian. Am 08.11.19 um 11:32 schrieb Deng, Emily: Hi Christian, The drm_sched_job_timedout-> amdgpu_job_timedout call amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in amdgpu_device_gpu_recover, and before calling drm_sched_stop. Best wishes Emily Deng -Original Message- From: Koenig, Christian Sent: Friday, November 8, 2019 6:26 PM To: Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Emily, well who is calling amdgpu_device_gpu_recover() in this case? When it's not the scheduler we shouldn't have a guilty job in the first place. Regards, Christian. Am 08.11.19 um 11:22 schrieb Deng, Emily: Hi Chrisitan, No, I am with the new branch and also has the patch. Even it are freed by main scheduler, how we could avoid main scheduler to free jobs while enter to function amdgpu_device_gpu_recover? Best wishes Emily Deng -Original Message- From: Koenig, Christian Sent: Friday, November 8, 2019 6:15 PM To: Deng, Emily ; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr Hi Emily, in this case you are on an old code branch. Jobs are freed now by the main scheduler thread and only if no timeout handler is running. See this patch here: commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e Author: Christian König Date: Thu Apr 18 11:00:21 2019 -0400 drm/scheduler: rework job destruction Regards, Christian. Am 08.11.19 um 11:11 schrieb Deng, Emily: Hi Christian, Please refer to follow log, when it enter to amdgpu_device_gpu_recover function, the bad job 5086879e is freeing in function amdgpu_job_free_cb at the same time, because of the hardware fence signal. But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is already freed, but job is not freed in time. Then this issue occurs. [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794163] amdgpu :00:08.0: GPU reset begin! [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794221] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:66eb74ab [ 449.794222] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:d4438ad9 [ 449.794255] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:b6d69c65 [ 449.794257] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:ea85e922 [ 449.794287] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread p
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Christian and Andrey, The issue I encountered is the bad job is freeing after entering to the amdgpu_device_gpu_recover. Don't know why, as per Christian said, it will call cancel_delayed_work in drm_sched_cleanup_jobs. Best wishes Emily Deng >-Original Message- >From: amd-gfx On Behalf Of Deng, >Emily >Sent: Monday, November 11, 2019 3:19 PM >To: Grodzovsky, Andrey ; Koenig, Christian >; amd-gfx@lists.freedesktop.org >Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Andrey, >I don’t think your patch will help for this. As it will may call >kthread_should_park in drm_sched_cleanup_jobs first, and then call >kcl_kthread_park. And then it still has a race between the 2 threads. > >Best wishes >Emily Deng > > > >>-Original Message- >>From: Grodzovsky, Andrey >>Sent: Saturday, November 9, 2019 3:01 AM >>To: Koenig, Christian ; Deng, Emily >>; amd-gfx@lists.freedesktop.org >>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> >>On 11/8/19 5:35 AM, Koenig, Christian wrote: >>> Hi Emily, >>> >>> exactly that can't happen. See here: >>> >>>> /* Don't destroy jobs while the timeout worker is running >>>> */ >>>> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>>> !cancel_delayed_work(&sched->work_tdr)) >>>> return NULL; >>> We never free jobs while the timeout working is running to prevent >>> exactly that issue. >> >> >>I don't think this protects us if drm_sched_cleanup_jobs is called for >>scheduler which didn't experience a timeout, in >>amdgpu_device_gpu_recover we access >>sched->ring_mirror_list for all the schedulers on a device so this >>sched->condition >>above won't protect us. What in fact could help maybe is my recent >>patch >>541c521 drm/sched: Avoid job cleanup if sched thread is parked. because >>we do park each of the scheduler threads during tdr job before trying >>to access >>sched->ring_mirror_list. >> >>Emily - did you see this problem with that patch in place ? I only >>pushed it yesterday. >> >>Andrey >> >> >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:32 schrieb Deng, Emily: >>>> Hi Christian, >>>> The drm_sched_job_timedout-> amdgpu_job_timedout call >>amdgpu_device_gpu_recover. I mean the main scheduler free the jobs >>while in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >>>> >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:26 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> well who is calling amdgpu_device_gpu_recover() in this case? >>>>> >>>>> When it's not the scheduler we shouldn't have a guilty job in the first >place. >>>>> >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>>>> Hi Chrisitan, >>>>>> No, I am with the new branch and also has the patch. Even >>>>>> it are freed by >>>>> main scheduler, how we could avoid main scheduler to free jobs >>>>> while enter to function amdgpu_device_gpu_recover? >>>>>> Best wishes >>>>>> Emily Deng >>>>>> >>>>>> >>>>>> >>>>>>> -Original Message- >>>>>>> From: Koenig, Christian >>>>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>>>> To: Deng, Emily ; >>>>>>> amd-gfx@lists.freedesktop.org >>>>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for >>>>>>> tdr >>>>>>> >>>>>>> Hi Emily, >>>>>>> >>>>>>> in this case you are on an old code branch. >>>>>>> >>>>>>> Jobs are freed now by the main scheduler thread and only if no >>>>>>> timeout
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Andrey, I don’t think your patch will help for this. As it will may call kthread_should_park in drm_sched_cleanup_jobs first, and then call kcl_kthread_park. And then it still has a race between the 2 threads. Best wishes Emily Deng >-Original Message- >From: Grodzovsky, Andrey >Sent: Saturday, November 9, 2019 3:01 AM >To: Koenig, Christian ; Deng, Emily >; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > > >On 11/8/19 5:35 AM, Koenig, Christian wrote: >> Hi Emily, >> >> exactly that can't happen. See here: >> >>> /* Don't destroy jobs while the timeout worker is running */ >>> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>> !cancel_delayed_work(&sched->work_tdr)) >>> return NULL; >> We never free jobs while the timeout working is running to prevent >> exactly that issue. > > >I don't think this protects us if drm_sched_cleanup_jobs is called for >scheduler >which didn't experience a timeout, in amdgpu_device_gpu_recover we access >sched->ring_mirror_list for all the schedulers on a device so this condition >above won't protect us. What in fact could help maybe is my recent patch >541c521 drm/sched: Avoid job cleanup if sched thread is parked. because we >do park each of the scheduler threads during tdr job before trying to access >sched->ring_mirror_list. > >Emily - did you see this problem with that patch in place ? I only pushed it >yesterday. > >Andrey > > >> >> Regards, >> Christian. >> >> Am 08.11.19 um 11:32 schrieb Deng, Emily: >>> Hi Christian, >>>The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >>> >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: Koenig, Christian >>>> Sent: Friday, November 8, 2019 6:26 PM >>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>> Hi Emily, >>>> >>>> well who is calling amdgpu_device_gpu_recover() in this case? >>>> >>>> When it's not the scheduler we shouldn't have a guilty job in the first >>>> place. >>>> >>>> Regards, >>>> Christian. >>>> >>>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>>> Hi Chrisitan, >>>>> No, I am with the new branch and also has the patch. Even >>>>> it are freed by >>>> main scheduler, how we could avoid main scheduler to free jobs while >>>> enter to function amdgpu_device_gpu_recover? >>>>> Best wishes >>>>> Emily Deng >>>>> >>>>> >>>>> >>>>>> -Original Message- >>>>>> From: Koenig, Christian >>>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>>> To: Deng, Emily ; >>>>>> amd-gfx@lists.freedesktop.org >>>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for >>>>>> tdr >>>>>> >>>>>> Hi Emily, >>>>>> >>>>>> in this case you are on an old code branch. >>>>>> >>>>>> Jobs are freed now by the main scheduler thread and only if no >>>>>> timeout handler is running. >>>>>> >>>>>> See this patch here: >>>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>>> Author: Christian König >>>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>>> >>>>>>> drm/scheduler: rework job destruction >>>>>> Regards, >>>>>> Christian. >>>>>> >>>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>>> Hi Christian, >>>>>>> Please refer to follow log, when it enter to >>>>>>> amdgpu_device_gpu_recover >>>>>> function, the bad job 5086879e is freeing in function >>>>>> amdgpu_job_free_cb at the same time, because of the hardware >>>>>> fence >>>> signal. >>>>>
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
On 11/8/19 5:54 AM, Deng, Emily wrote: > Hi Christian, > Sorry, seems I understand wrong. And from the print, the free job's > thread is the same as job timeout thread. So seems have some issue in > function amdgpu_device_gpu_recover. I don't think it's correct, seems your prints just don't print the pids because it's all zeros which cannot be true for any kernel or user mode thread. In fact looking at the OOPs i see the actual TDR job pid which is 55 CPU: 3 PID: 55 Comm: kworker/3:1 Tainted Andrey > > > Best wishes > Emily Deng > > > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 6:35 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Hi Emily, >> >> exactly that can't happen. See here: >> >>> /* Don't destroy jobs while the timeout worker is running */ >>> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >>> !cancel_delayed_work(&sched->work_tdr)) >>> return NULL; >> We never free jobs while the timeout working is running to prevent exactly >> that issue. >> >> Regards, >> Christian. >> >> Am 08.11.19 um 11:32 schrieb Deng, Emily: >>> Hi Christian, >>>The drm_sched_job_timedout-> amdgpu_job_timedout call >> amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >> in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: Koenig, Christian >>>> Sent: Friday, November 8, 2019 6:26 PM >>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>> Hi Emily, >>>> >>>> well who is calling amdgpu_device_gpu_recover() in this case? >>>> >>>> When it's not the scheduler we shouldn't have a guilty job in the first >>>> place. >>>> >>>> Regards, >>>> Christian. >>>> >>>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>>> Hi Chrisitan, >>>>> No, I am with the new branch and also has the patch. Even it >>>>> are freed by >>>> main scheduler, how we could avoid main scheduler to free jobs while >>>> enter to function amdgpu_device_gpu_recover? >>>>> Best wishes >>>>> Emily Deng >>>>> >>>>> >>>>> >>>>>> -Original Message- >>>>>> From: Koenig, Christian >>>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>>> To: Deng, Emily ; amd- >> g...@lists.freedesktop.org >>>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>>> >>>>>> Hi Emily, >>>>>> >>>>>> in this case you are on an old code branch. >>>>>> >>>>>> Jobs are freed now by the main scheduler thread and only if no >>>>>> timeout handler is running. >>>>>> >>>>>> See this patch here: >>>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>>> Author: Christian König >>>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>>> >>>>>>> drm/scheduler: rework job destruction >>>>>> Regards, >>>>>> Christian. >>>>>> >>>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>>> Hi Christian, >>>>>>> Please refer to follow log, when it enter to >>>>>>> amdgpu_device_gpu_recover >>>>>> function, the bad job 5086879e is freeing in function >>>>>> amdgpu_job_free_cb at the same time, because of the hardware fence >>>> signal. >>>>>> But amdgpu_device_gpu_recover goes faster, at this case, the >>>>>> s_fence is already freed, but job is not freed in time. Then this issue >> occurs. >>>>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >>>> sdma0 >>>>>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>>>>> [drm:amdgpu_job_
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
On 11/8/19 5:35 AM, Koenig, Christian wrote: > Hi Emily, > > exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >> !cancel_delayed_work(&sched->work_tdr)) >> return NULL; > We never free jobs while the timeout working is running to prevent > exactly that issue. I don't think this protects us if drm_sched_cleanup_jobs is called for scheduler which didn't experience a timeout, in amdgpu_device_gpu_recover we access sched->ring_mirror_list for all the schedulers on a device so this condition above won't protect us. What in fact could help maybe is my recent patch 541c521 drm/sched: Avoid job cleanup if sched thread is parked. because we do park each of the scheduler threads during tdr job before trying to access sched->ring_mirror_list. Emily - did you see this problem with that patch in place ? I only pushed it yesterday. Andrey > > Regards, > Christian. > > Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >>The drm_sched_job_timedout-> amdgpu_job_timedout call >> amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in >> amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -----Original Message----- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first >>> place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>> No, I am with the new branch and also has the patch. Even it are >>>> freed by >>> main scheduler, how we could avoid main scheduler to free jobs while enter >>> to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Author: Christian König >>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>> >>>>>> drm/scheduler: rework job destruction >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> Please refer to follow log, when it enter to >>>>>> amdgpu_device_gpu_recover >>>>> function, the bad job 5086879e is freeing in function >>>>> amdgpu_job_free_cb at the same time, because of the hardware fence >>> signal. >>>>> But amdgpu_device_gpu_recover goes faster, at this case, the s_fence >>>>> is already freed, but job is not freed in time. Then this issue occurs. >>>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >>> sdma0 >>>>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>>>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >>>>> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] >>>>> amdgpu >>>>> :00:08.0: GPU reset begin! >>>>>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >>>>>> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>>>> pid 0, s_job:66eb74ab [ 449.794222] >>
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Christian, Sorry, seems I understand wrong. And from the print, the free job's thread is the same as job timeout thread. So seems have some issue in function amdgpu_device_gpu_recover. Best wishes Emily Deng >-Original Message- >From: Koenig, Christian >Sent: Friday, November 8, 2019 6:35 PM >To: Deng, Emily ; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >exactly that can't happen. See here: > >> /* Don't destroy jobs while the timeout worker is running */ >> if (sched->timeout != MAX_SCHEDULE_TIMEOUT && >> !cancel_delayed_work(&sched->work_tdr)) >> return NULL; > >We never free jobs while the timeout working is running to prevent exactly >that issue. > >Regards, >Christian. > >Am 08.11.19 um 11:32 schrieb Deng, Emily: >> Hi Christian, >> The drm_sched_job_timedout-> amdgpu_job_timedout call >amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while >in amdgpu_device_gpu_recover, and before calling drm_sched_stop. >> >> Best wishes >> Emily Deng >> >> >> >>> -----Original Message----- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:26 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> well who is calling amdgpu_device_gpu_recover() in this case? >>> >>> When it's not the scheduler we shouldn't have a guilty job in the first >>> place. >>> >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>>> Hi Chrisitan, >>>>No, I am with the new branch and also has the patch. Even it >>>> are freed by >>> main scheduler, how we could avoid main scheduler to free jobs while >>> enter to function amdgpu_device_gpu_recover? >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: Koenig, Christian >>>>> Sent: Friday, November 8, 2019 6:15 PM >>>>> To: Deng, Emily ; amd- >g...@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Hi Emily, >>>>> >>>>> in this case you are on an old code branch. >>>>> >>>>> Jobs are freed now by the main scheduler thread and only if no >>>>> timeout handler is running. >>>>> >>>>> See this patch here: >>>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>>> Author: Christian König >>>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>>> >>>>>> drm/scheduler: rework job destruction >>>>> Regards, >>>>> Christian. >>>>> >>>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>>> Hi Christian, >>>>>> Please refer to follow log, when it enter to >>>>>> amdgpu_device_gpu_recover >>>>> function, the bad job 5086879e is freeing in function >>>>> amdgpu_job_free_cb at the same time, because of the hardware fence >>> signal. >>>>> But amdgpu_device_gpu_recover goes faster, at this case, the >>>>> s_fence is already freed, but job is not freed in time. Then this issue >occurs. >>>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >>> sdma0 >>>>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>>>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >>>>> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] >>>>> amdgpu >>>>> :00:08.0: GPU reset begin! >>>>>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: >>>>>> process pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 >>>>>> thread pid 0, s_job:66eb74ab [ 449.794222] >>>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 >>>>>> thread pid 0, s_job:d4438ad9 [ 449.794255] >>>>>> Emily:amdgpu_job_free_cb,Pro
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Emily, exactly that can't happen. See here: > /* Don't destroy jobs while the timeout worker is running */ > if (sched->timeout != MAX_SCHEDULE_TIMEOUT && > !cancel_delayed_work(&sched->work_tdr)) > return NULL; We never free jobs while the timeout working is running to prevent exactly that issue. Regards, Christian. Am 08.11.19 um 11:32 schrieb Deng, Emily: > Hi Christian, > The drm_sched_job_timedout-> amdgpu_job_timedout call > amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in > amdgpu_device_gpu_recover, and before calling drm_sched_stop. > > Best wishes > Emily Deng > > > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 6:26 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Hi Emily, >> >> well who is calling amdgpu_device_gpu_recover() in this case? >> >> When it's not the scheduler we shouldn't have a guilty job in the first >> place. >> >> Regards, >> Christian. >> >> Am 08.11.19 um 11:22 schrieb Deng, Emily: >>> Hi Chrisitan, >>>No, I am with the new branch and also has the patch. Even it are >>> freed by >> main scheduler, how we could avoid main scheduler to free jobs while enter >> to function amdgpu_device_gpu_recover? >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: Koenig, Christian >>>> Sent: Friday, November 8, 2019 6:15 PM >>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>> Hi Emily, >>>> >>>> in this case you are on an old code branch. >>>> >>>> Jobs are freed now by the main scheduler thread and only if no >>>> timeout handler is running. >>>> >>>> See this patch here: >>>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>>> Author: Christian König >>>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>>> >>>>> drm/scheduler: rework job destruction >>>> Regards, >>>> Christian. >>>> >>>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>>> Hi Christian, >>>>> Please refer to follow log, when it enter to >>>>> amdgpu_device_gpu_recover >>>> function, the bad job 5086879e is freeing in function >>>> amdgpu_job_free_cb at the same time, because of the hardware fence >> signal. >>>> But amdgpu_device_gpu_recover goes faster, at this case, the s_fence >>>> is already freed, but job is not freed in time. Then this issue occurs. >>>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >> sdma0 >>>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >>>> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] >>>> amdgpu >>>> :00:08.0: GPU reset begin! >>>>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >>>>> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>>> pid 0, s_job:66eb74ab [ 449.794222] >>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>>> pid 0, s_job:d4438ad9 [ 449.794255] >>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>>> pid 0, s_job:b6d69c65 [ 449.794257] >>>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>>> pid 0, >>>> s_job:ea85e922 [ 449.794287] >>>> Emily:amdgpu_job_free_cb,Process >>>> information: process pid 0 thread pid 0, s_job:ed3a5ac6 [ >>>> 449.794366] BUG: unable to handle kernel NULL pointer dereference at >>>> 00c0 [ 449.800818] PGD 0 P4D 0 [ 449.801040] Oops: >>>> [#1] SMP PTI >>>>> [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE >>>> 4.18.0-15-generic #16~18.04.1-Ubuntu >>>>> [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX,
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Christian, The drm_sched_job_timedout-> amdgpu_job_timedout call amdgpu_device_gpu_recover. I mean the main scheduler free the jobs while in amdgpu_device_gpu_recover, and before calling drm_sched_stop. Best wishes Emily Deng >-Original Message- >From: Koenig, Christian >Sent: Friday, November 8, 2019 6:26 PM >To: Deng, Emily ; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >well who is calling amdgpu_device_gpu_recover() in this case? > >When it's not the scheduler we shouldn't have a guilty job in the first place. > >Regards, >Christian. > >Am 08.11.19 um 11:22 schrieb Deng, Emily: >> Hi Chrisitan, >> No, I am with the new branch and also has the patch. Even it are freed >> by >main scheduler, how we could avoid main scheduler to free jobs while enter >to function amdgpu_device_gpu_recover? >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 6:15 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Hi Emily, >>> >>> in this case you are on an old code branch. >>> >>> Jobs are freed now by the main scheduler thread and only if no >>> timeout handler is running. >>> >>> See this patch here: >>>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>>> Author: Christian König >>>> Date: Thu Apr 18 11:00:21 2019 -0400 >>>> >>>> drm/scheduler: rework job destruction >>> Regards, >>> Christian. >>> >>> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>>> Hi Christian, >>>>Please refer to follow log, when it enter to >>>> amdgpu_device_gpu_recover >>> function, the bad job 5086879e is freeing in function >>> amdgpu_job_free_cb at the same time, because of the hardware fence >signal. >>> But amdgpu_device_gpu_recover goes faster, at this case, the s_fence >>> is already freed, but job is not freed in time. Then this issue occurs. >>>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring >sdma0 >>>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >>> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] >>> amdgpu >>> :00:08.0: GPU reset begin! >>>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >>>> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>> pid 0, s_job:66eb74ab [ 449.794222] >>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>> pid 0, s_job:d4438ad9 [ 449.794255] >>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>> pid 0, s_job:b6d69c65 [ 449.794257] >>>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>>> pid 0, >>> s_job:ea85e922 [ 449.794287] >>> Emily:amdgpu_job_free_cb,Process >>> information: process pid 0 thread pid 0, s_job:ed3a5ac6 [ >>> 449.794366] BUG: unable to handle kernel NULL pointer dereference at >>> 00c0 [ 449.800818] PGD 0 P4D 0 [ 449.801040] Oops: >>> [#1] SMP PTI >>>> [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE >>> 4.18.0-15-generic #16~18.04.1-Ubuntu >>>> [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX, >>>> 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 449.802944] >>>> Workqueue: events drm_sched_job_timedout [amd_sched] [ 449.803488] >RIP: >>> 0010:amdgpu_device_gpu_recover+0x1da/0xb60 [amdgpu] >>>> [ 449.804020] Code: dd ff ff 49 39 c5 48 89 55 a8 0f 85 56 ff ff ff >>>> 45 85 e4 0f >>> 85 a1 00 00 00 48 8b 45 b0 48 85 c0 0f 84 60 01 00 00 48 8b 40 10 <48> 8b >98 >>> c0 00 00 00 48 85 db 0f 84 4c 01 00 00 48 8b 43 48 a8 01 >>>> [ 449.805593] RSP: 0018:b4c7c08f7d68 EFLAGS: 00010286 [ >>>> 449.806032] RAX: RBX: RCX: >>>> [ 449.806625] RDX: b4c7c08f5ac0 RSI: >>>> 000fffe0 RDI: 0246 [ 449.807224] RBP: >>>>
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Emily, well who is calling amdgpu_device_gpu_recover() in this case? When it's not the scheduler we shouldn't have a guilty job in the first place. Regards, Christian. Am 08.11.19 um 11:22 schrieb Deng, Emily: > Hi Chrisitan, > No, I am with the new branch and also has the patch. Even it are freed > by main scheduler, how we could avoid main scheduler to free jobs while enter > to function amdgpu_device_gpu_recover? > > Best wishes > Emily Deng > > > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 6:15 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Hi Emily, >> >> in this case you are on an old code branch. >> >> Jobs are freed now by the main scheduler thread and only if no timeout >> handler is running. >> >> See this patch here: >>> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >>> Author: Christian König >>> Date: Thu Apr 18 11:00:21 2019 -0400 >>> >>> drm/scheduler: rework job destruction >> Regards, >> Christian. >> >> Am 08.11.19 um 11:11 schrieb Deng, Emily: >>> Hi Christian, >>>Please refer to follow log, when it enter to >>> amdgpu_device_gpu_recover >> function, the bad job 5086879e is freeing in function >> amdgpu_job_free_cb at the same time, because of the hardware fence signal. >> But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is >> already freed, but job is not freed in time. Then this issue occurs. >>> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 >>> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >>> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >> process pid 0 thread pid 0, s_job:5086879e [ 449.794163] amdgpu >> :00:08.0: GPU reset begin! >>> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >>> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>> pid 0, s_job:66eb74ab [ 449.794222] >>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>> pid 0, s_job:d4438ad9 [ 449.794255] >>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >>> pid 0, s_job:b6d69c65 [ 449.794257] >>> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, >> s_job:ea85e922 [ 449.794287] Emily:amdgpu_job_free_cb,Process >> information: process pid 0 thread pid 0, s_job:ed3a5ac6 >> [ 449.794366] BUG: unable to handle kernel NULL pointer dereference at >> 00c0 [ 449.800818] PGD 0 P4D 0 [ 449.801040] Oops: >> [#1] SMP PTI >>> [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE >> 4.18.0-15-generic #16~18.04.1-Ubuntu >>> [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >>> BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 449.802944] Workqueue: events >>> drm_sched_job_timedout [amd_sched] [ 449.803488] RIP: >> 0010:amdgpu_device_gpu_recover+0x1da/0xb60 [amdgpu] >>> [ 449.804020] Code: dd ff ff 49 39 c5 48 89 55 a8 0f 85 56 ff ff ff 45 85 >>> e4 0f >> 85 a1 00 00 00 48 8b 45 b0 48 85 c0 0f 84 60 01 00 00 48 8b 40 10 <48> 8b 98 >> c0 00 00 00 48 85 db 0f 84 4c 01 00 00 48 8b 43 48 a8 01 >>> [ 449.805593] RSP: 0018:b4c7c08f7d68 EFLAGS: 00010286 [ >>> 449.806032] RAX: RBX: RCX: >>> [ 449.806625] RDX: b4c7c08f5ac0 RSI: >>> 000fffe0 RDI: 0246 [ 449.807224] RBP: >>> b4c7c08f7de0 R08: 0068b9d54000 R09: [ >>> 449.807818] R10: R11: 0148 R12: >>> [ 449.808411] R13: b4c7c08f7da0 R14: >>> 8d82b8525d40 R15: 8d82b8525d40 [ 449.809004] FS: >>> () GS:8d82bfd8() >>> knlGS: [ 449.809674] CS: 0010 DS: ES: CR0: >>> 80050033 [ 449.810153] CR2: 00c0 CR3: >>> 3cc0a001 CR4: 003606e0 [ 449.810747] DR0: >> DR1: DR2: >> [ 449.811344] DR3: DR6: fffe0ff0 DR7: >> 0400 [ 449.811937] Call Trace: >>> [ 449.812206] amdgpu_job_timedout+0x114/0x140 [amdgpu] [ >
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Chrisitan, No, I am with the new branch and also has the patch. Even it are freed by main scheduler, how we could avoid main scheduler to free jobs while enter to function amdgpu_device_gpu_recover? Best wishes Emily Deng >-Original Message- >From: Koenig, Christian >Sent: Friday, November 8, 2019 6:15 PM >To: Deng, Emily ; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Hi Emily, > >in this case you are on an old code branch. > >Jobs are freed now by the main scheduler thread and only if no timeout >handler is running. > >See this patch here: >> commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e >> Author: Christian König >> Date: Thu Apr 18 11:00:21 2019 -0400 >> >> drm/scheduler: rework job destruction > >Regards, >Christian. > >Am 08.11.19 um 11:11 schrieb Deng, Emily: >> Hi Christian, >> Please refer to follow log, when it enter to amdgpu_device_gpu_recover >function, the bad job 5086879e is freeing in function >amdgpu_job_free_cb at the same time, because of the hardware fence signal. >But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is >already freed, but job is not freed in time. Then this issue occurs. >> >> [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 >> timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] >> [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: >process pid 0 thread pid 0, s_job:5086879e [ 449.794163] amdgpu >:00:08.0: GPU reset begin! >> [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process >> pid 0 thread pid 0, s_job:5086879e [ 449.794221] >> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >> pid 0, s_job:66eb74ab [ 449.794222] >> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >> pid 0, s_job:d4438ad9 [ 449.794255] >> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread >> pid 0, s_job:b6d69c65 [ 449.794257] >> Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, >s_job:ea85e922 [ 449.794287] Emily:amdgpu_job_free_cb,Process >information: process pid 0 thread pid 0, s_job:ed3a5ac6 >[ 449.794366] BUG: unable to handle kernel NULL pointer dereference at >00c0 [ 449.800818] PGD 0 P4D 0 [ 449.801040] Oops: >[#1] SMP PTI >> [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE >4.18.0-15-generic #16~18.04.1-Ubuntu >> [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 449.802944] Workqueue: events >> drm_sched_job_timedout [amd_sched] [ 449.803488] RIP: >0010:amdgpu_device_gpu_recover+0x1da/0xb60 [amdgpu] >> [ 449.804020] Code: dd ff ff 49 39 c5 48 89 55 a8 0f 85 56 ff ff ff 45 85 >> e4 0f >85 a1 00 00 00 48 8b 45 b0 48 85 c0 0f 84 60 01 00 00 48 8b 40 10 <48> 8b 98 >c0 00 00 00 48 85 db 0f 84 4c 01 00 00 48 8b 43 48 a8 01 >> [ 449.805593] RSP: 0018:b4c7c08f7d68 EFLAGS: 00010286 [ >> 449.806032] RAX: RBX: RCX: >> [ 449.806625] RDX: b4c7c08f5ac0 RSI: >> 000fffe0 RDI: 0246 [ 449.807224] RBP: >> b4c7c08f7de0 R08: 0068b9d54000 R09: [ >> 449.807818] R10: R11: 0148 R12: >> [ 449.808411] R13: b4c7c08f7da0 R14: >> 8d82b8525d40 R15: 8d82b8525d40 [ 449.809004] FS: >> () GS:8d82bfd8() >> knlGS: [ 449.809674] CS: 0010 DS: ES: CR0: >> 80050033 [ 449.810153] CR2: 00c0 CR3: >> 3cc0a001 CR4: 003606e0 [ 449.810747] DR0: > DR1: DR2: >[ 449.811344] DR3: DR6: fffe0ff0 DR7: >0400 [ 449.811937] Call Trace: >> [ 449.812206] amdgpu_job_timedout+0x114/0x140 [amdgpu] [ >> 449.812635] drm_sched_job_timedout+0x44/0x90 [amd_sched] [ >> 449.813139] ? amdgpu_cgs_destroy_device+0x10/0x10 [amdgpu] [ >> 449.813609] ? drm_sched_job_timedout+0x44/0x90 [amd_sched] [ >> 449.814077] process_one_work+0x1fd/0x3f0 [ 449.814417] >> worker_thread+0x34/0x410 [ 449.814728] kthread+0x121/0x140 [ >> 449.815004] ? process_one_work+0x3f0/0x3f0 [ 449.815374] ? >> kthread_create_worker_on_cpu+0x70/0x70 >> [ 449.815799] ret_from_fork+0x35/0x40 >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Fr
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Emily, in this case you are on an old code branch. Jobs are freed now by the main scheduler thread and only if no timeout handler is running. See this patch here: > commit 5918045c4ed492fb5813f980dcf89a90fefd0a4e > Author: Christian König > Date: Thu Apr 18 11:00:21 2019 -0400 > > drm/scheduler: rework job destruction Regards, Christian. Am 08.11.19 um 11:11 schrieb Deng, Emily: > Hi Christian, > Please refer to follow log, when it enter to amdgpu_device_gpu_recover > function, the bad job 5086879e is freeing in function > amdgpu_job_free_cb at the same time, because of the hardware fence signal. > But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is > already freed, but job is not freed in time. Then this issue occurs. > > [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, > signaled seq=2481, emitted seq=2483 > [ 449.793202] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process > information: process pid 0 thread pid 0, s_job:5086879e > [ 449.794163] amdgpu :00:08.0: GPU reset begin! > [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:5086879e > [ 449.794221] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:66eb74ab > [ 449.794222] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:d4438ad9 > [ 449.794255] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:b6d69c65 > [ 449.794257] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:ea85e922 > [ 449.794287] Emily:amdgpu_job_free_cb,Process information: process pid 0 > thread pid 0, s_job:ed3a5ac6 > [ 449.794366] BUG: unable to handle kernel NULL pointer dereference at > 00c0 > [ 449.800818] PGD 0 P4D 0 > [ 449.801040] Oops: [#1] SMP PTI > [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE > 4.18.0-15-generic #16~18.04.1-Ubuntu > [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > Ubuntu-1.8.2-1ubuntu1 04/01/2014 > [ 449.802944] Workqueue: events drm_sched_job_timedout [amd_sched] > [ 449.803488] RIP: 0010:amdgpu_device_gpu_recover+0x1da/0xb60 [amdgpu] > [ 449.804020] Code: dd ff ff 49 39 c5 48 89 55 a8 0f 85 56 ff ff ff 45 85 e4 > 0f 85 a1 00 00 00 48 8b 45 b0 48 85 c0 0f 84 60 01 00 00 48 8b 40 10 <48> 8b > 98 c0 00 00 00 48 85 db 0f 84 4c 01 00 00 48 8b 43 48 a8 01 > [ 449.805593] RSP: 0018:b4c7c08f7d68 EFLAGS: 00010286 > [ 449.806032] RAX: RBX: RCX: > > [ 449.806625] RDX: b4c7c08f5ac0 RSI: 000fffe0 RDI: > 0246 > [ 449.807224] RBP: b4c7c08f7de0 R08: 0068b9d54000 R09: > > [ 449.807818] R10: R11: 0148 R12: > > [ 449.808411] R13: b4c7c08f7da0 R14: 8d82b8525d40 R15: > 8d82b8525d40 > [ 449.809004] FS: () GS:8d82bfd8() > knlGS: > [ 449.809674] CS: 0010 DS: ES: CR0: 80050033 > [ 449.810153] CR2: 00c0 CR3: 3cc0a001 CR4: > 003606e0 > [ 449.810747] DR0: DR1: DR2: > > [ 449.811344] DR3: DR6: fffe0ff0 DR7: > 0400 > [ 449.811937] Call Trace: > [ 449.812206] amdgpu_job_timedout+0x114/0x140 [amdgpu] > [ 449.812635] drm_sched_job_timedout+0x44/0x90 [amd_sched] > [ 449.813139] ? amdgpu_cgs_destroy_device+0x10/0x10 [amdgpu] > [ 449.813609] ? drm_sched_job_timedout+0x44/0x90 [amd_sched] > [ 449.814077] process_one_work+0x1fd/0x3f0 > [ 449.814417] worker_thread+0x34/0x410 > [ 449.814728] kthread+0x121/0x140 > [ 449.815004] ? process_one_work+0x3f0/0x3f0 > [ 449.815374] ? kthread_create_worker_on_cpu+0x70/0x70 > [ 449.815799] ret_from_fork+0x35/0x40 > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 5:43 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Am 08.11.19 um 10:39 schrieb Deng, Emily: >>> Sorry, please take your time. >> Have you seen my other response a bit below? >> >> I can't follow how it would be possible for job->s_fence to be NULL without >> the job also being freed. >> >> So it looks like this patch is just papering over some bigger issues. >> >> Regards, >> Christian. >> >>> Best wishe
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Hi Christian, Please refer to follow log, when it enter to amdgpu_device_gpu_recover function, the bad job 5086879e is freeing in function amdgpu_job_free_cb at the same time, because of the hardware fence signal. But amdgpu_device_gpu_recover goes faster, at this case, the s_fence is already freed, but job is not freed in time. Then this issue occurs. [ 449.792189] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=2481, emitted seq=2483 [ 449.793202] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794163] amdgpu :00:08.0: GPU reset begin! [ 449.794175] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:5086879e [ 449.794221] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:66eb74ab [ 449.794222] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:d4438ad9 [ 449.794255] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:b6d69c65 [ 449.794257] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:ea85e922 [ 449.794287] Emily:amdgpu_job_free_cb,Process information: process pid 0 thread pid 0, s_job:ed3a5ac6 [ 449.794366] BUG: unable to handle kernel NULL pointer dereference at 00c0 [ 449.800818] PGD 0 P4D 0 [ 449.801040] Oops: [#1] SMP PTI [ 449.801338] CPU: 3 PID: 55 Comm: kworker/3:1 Tainted: G OE 4.18.0-15-generic #16~18.04.1-Ubuntu [ 449.802157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 [ 449.802944] Workqueue: events drm_sched_job_timedout [amd_sched] [ 449.803488] RIP: 0010:amdgpu_device_gpu_recover+0x1da/0xb60 [amdgpu] [ 449.804020] Code: dd ff ff 49 39 c5 48 89 55 a8 0f 85 56 ff ff ff 45 85 e4 0f 85 a1 00 00 00 48 8b 45 b0 48 85 c0 0f 84 60 01 00 00 48 8b 40 10 <48> 8b 98 c0 00 00 00 48 85 db 0f 84 4c 01 00 00 48 8b 43 48 a8 01 [ 449.805593] RSP: 0018:b4c7c08f7d68 EFLAGS: 00010286 [ 449.806032] RAX: RBX: RCX: [ 449.806625] RDX: b4c7c08f5ac0 RSI: 000fffe0 RDI: 0246 [ 449.807224] RBP: b4c7c08f7de0 R08: 0068b9d54000 R09: [ 449.807818] R10: R11: 0148 R12: [ 449.808411] R13: b4c7c08f7da0 R14: 8d82b8525d40 R15: 8d82b8525d40 [ 449.809004] FS: () GS:8d82bfd8() knlGS: [ 449.809674] CS: 0010 DS: ES: CR0: 80050033 [ 449.810153] CR2: 00c0 CR3: 3cc0a001 CR4: 003606e0 [ 449.810747] DR0: DR1: DR2: [ 449.811344] DR3: DR6: fffe0ff0 DR7: 0400 [ 449.811937] Call Trace: [ 449.812206] amdgpu_job_timedout+0x114/0x140 [amdgpu] [ 449.812635] drm_sched_job_timedout+0x44/0x90 [amd_sched] [ 449.813139] ? amdgpu_cgs_destroy_device+0x10/0x10 [amdgpu] [ 449.813609] ? drm_sched_job_timedout+0x44/0x90 [amd_sched] [ 449.814077] process_one_work+0x1fd/0x3f0 [ 449.814417] worker_thread+0x34/0x410 [ 449.814728] kthread+0x121/0x140 [ 449.815004] ? process_one_work+0x3f0/0x3f0 [ 449.815374] ? kthread_create_worker_on_cpu+0x70/0x70 [ 449.815799] ret_from_fork+0x35/0x40 >-Original Message- >From: Koenig, Christian >Sent: Friday, November 8, 2019 5:43 PM >To: Deng, Emily ; amd-gfx@lists.freedesktop.org >Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr > >Am 08.11.19 um 10:39 schrieb Deng, Emily: >> Sorry, please take your time. > >Have you seen my other response a bit below? > >I can't follow how it would be possible for job->s_fence to be NULL without >the job also being freed. > >So it looks like this patch is just papering over some bigger issues. > >Regards, >Christian. > >> >> Best wishes >> Emily Deng >> >> >> >>> -Original Message- >>> From: Koenig, Christian >>> Sent: Friday, November 8, 2019 5:08 PM >>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>> >>> Am 08.11.19 um 09:52 schrieb Deng, Emily: >>>> Ping. >>> You need to give me at least enough time to wake up :) >>> >>>> >>>> Best wishes >>>> Emily Deng >>>> >>>> >>>> >>>>> -Original Message- >>>>> From: amd-gfx On Behalf Of >>>>> Deng, Emily >>>>> Sent: Friday, November 8, 2019 10:56 AM >>>>> To:
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Am 08.11.19 um 10:39 schrieb Deng, Emily: > Sorry, please take your time. Have you seen my other response a bit below? I can't follow how it would be possible for job->s_fence to be NULL without the job also being freed. So it looks like this patch is just papering over some bigger issues. Regards, Christian. > > Best wishes > Emily Deng > > > >> -Original Message- >> From: Koenig, Christian >> Sent: Friday, November 8, 2019 5:08 PM >> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >> >> Am 08.11.19 um 09:52 schrieb Deng, Emily: >>> Ping. >> You need to give me at least enough time to wake up :) >> >>> >>> Best wishes >>> Emily Deng >>> >>> >>> >>>> -Original Message- >>>> From: amd-gfx On Behalf Of >>>> Deng, Emily >>>> Sent: Friday, November 8, 2019 10:56 AM >>>> To: Koenig, Christian ; amd- >>>> g...@lists.freedesktop.org >>>> Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>> >>>>> -Original Message- >>>>> From: Christian König >>>>> Sent: Thursday, November 7, 2019 7:28 PM >>>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org >>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr >>>>> >>>>> Am 07.11.19 um 11:25 schrieb Emily Deng: >>>>>> When the job is already signaled, the s_fence is freed. Then it >>>>>> will has null pointer in amdgpu_device_gpu_recover. >>>>> NAK, the s_fence is only set to NULL when the job is destroyed. See >>>>> drm_sched_job_cleanup(). >>>> I know it is set to NULL in drm_sched_job_cleanup. But in one case, >>>> when it enter into the amdgpu_device_gpu_recover, it already in >>>> drm_sched_job_cleanup, and at this time, it will go to free job. But >>>> the amdgpu_device_gpu_recover sometimes is faster. At that time, job >>>> is not freed, but s_fence is already NULL. >> No, that case can't happen. See here: >> >>> drm_sched_job_cleanup(s_job); >>> >>> amdgpu_ring_priority_put(ring, s_job->s_priority); >>> dma_fence_put(job->fence); >>> amdgpu_sync_free(&job->sync); >>> amdgpu_sync_free(&job->sched_sync); >>> kfree(job); >> The job itself is freed up directly after freeing the reference to the >> s_fence. >> >> So you are just papering over a much bigger problem here. This patch is a >> clear NAK. >> >> Regards, >> Christian. >> >>>>> When you see a job without an s_fence then that means the problem is >>>>> somewhere else. >>>>> >>>>> Regards, >>>>> Christian. >>>>> >>>>>> Signed-off-by: Emily Deng >>>>>> --- >>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- >>>>>> drivers/gpu/drm/scheduler/sched_main.c | 11 ++- >>>>>> 2 files changed, 7 insertions(+), 6 deletions(-) >>>>>> >>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> index e6ce949..5a8f08e 100644 >>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>>>> @@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct >>>>> amdgpu_device *adev, >>>>>> * >>>>>> * job->base holds a reference to parent fence >>>>>> */ >>>>>> -if (job && job->base.s_fence->parent && >>>>>> +if (job && job->base.s_fence && job->base.s_fence->parent && >>>>>> dma_fence_is_signaled(job->base.s_fence->parent)) >>>>>> job_signaled = true; >>>>>> >>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>>>>> b/drivers/gpu/drm/scheduler/sched_main.c >>>>>> index 31809ca..56cc10e 100644 >>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Sorry, please take your time.
Best wishes
Emily Deng
>-Original Message-
>From: Koenig, Christian
>Sent: Friday, November 8, 2019 5:08 PM
>To: Deng, Emily ; amd-gfx@lists.freedesktop.org
>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>Am 08.11.19 um 09:52 schrieb Deng, Emily:
>> Ping.
>
>You need to give me at least enough time to wake up :)
>
>>
>>
>> Best wishes
>> Emily Deng
>>
>>
>>
>>> -Original Message-
>>> From: amd-gfx On Behalf Of
>>> Deng, Emily
>>> Sent: Friday, November 8, 2019 10:56 AM
>>> To: Koenig, Christian ; amd-
>>> g...@lists.freedesktop.org
>>> Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>>>
>>>> -Original Message-
>>>> From: Christian König
>>>> Sent: Thursday, November 7, 2019 7:28 PM
>>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org
>>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>>>>
>>>> Am 07.11.19 um 11:25 schrieb Emily Deng:
>>>>> When the job is already signaled, the s_fence is freed. Then it
>>>>> will has null pointer in amdgpu_device_gpu_recover.
>>>> NAK, the s_fence is only set to NULL when the job is destroyed. See
>>>> drm_sched_job_cleanup().
>>> I know it is set to NULL in drm_sched_job_cleanup. But in one case,
>>> when it enter into the amdgpu_device_gpu_recover, it already in
>>> drm_sched_job_cleanup, and at this time, it will go to free job. But
>>> the amdgpu_device_gpu_recover sometimes is faster. At that time, job
>>> is not freed, but s_fence is already NULL.
>
>No, that case can't happen. See here:
>
>> drm_sched_job_cleanup(s_job);
>>
>> amdgpu_ring_priority_put(ring, s_job->s_priority);
>> dma_fence_put(job->fence);
>> amdgpu_sync_free(&job->sync);
>> amdgpu_sync_free(&job->sched_sync);
>> kfree(job);
>
>The job itself is freed up directly after freeing the reference to the s_fence.
>
>So you are just papering over a much bigger problem here. This patch is a
>clear NAK.
>
>Regards,
>Christian.
>
>>>> When you see a job without an s_fence then that means the problem is
>>>> somewhere else.
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>> Signed-off-by: Emily Deng
>>>>> ---
>>>>>drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>>>>>drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
>>>>>2 files changed, 7 insertions(+), 6 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> index e6ce949..5a8f08e 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>>> @@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct
>>>> amdgpu_device *adev,
>>>>>*
>>>>>* job->base holds a reference to parent fence
>>>>>*/
>>>>> - if (job && job->base.s_fence->parent &&
>>>>> + if (job && job->base.s_fence && job->base.s_fence->parent &&
>>>>> dma_fence_is_signaled(job->base.s_fence->parent))
>>>>> job_signaled = true;
>>>>>
>>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>>>> index 31809ca..56cc10e 100644
>>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>>>> @@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct
>>>> drm_sched_job
>>>>> *bad)
>>>>>
>>>>> spin_lock(&rq->lock);
>>>>> list_for_each_entry_safe(entity, tmp,
>>>>> &rq->entities,
>>>> list) {
>>>>> - if (bad->s_fence->scheduled.context ==
>>>>> - entity->fence_context) {
>>>>> + if (bad->s_fence && (bad->s_fence-
>>>>> scheduled.cont
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Am 08.11.19 um 09:52 schrieb Deng, Emily:
> Ping.
You need to give me at least enough time to wake up :)
>
>
> Best wishes
> Emily Deng
>
>
>
>> -Original Message-
>> From: amd-gfx On Behalf Of Deng,
>> Emily
>> Sent: Friday, November 8, 2019 10:56 AM
>> To: Koenig, Christian ; amd-
>> g...@lists.freedesktop.org
>> Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>>
>>> -Original Message-
>>> From: Christian König
>>> Sent: Thursday, November 7, 2019 7:28 PM
>>> To: Deng, Emily ; amd-gfx@lists.freedesktop.org
>>> Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>>>
>>> Am 07.11.19 um 11:25 schrieb Emily Deng:
>>>> When the job is already signaled, the s_fence is freed. Then it will
>>>> has null pointer in amdgpu_device_gpu_recover.
>>> NAK, the s_fence is only set to NULL when the job is destroyed. See
>>> drm_sched_job_cleanup().
>> I know it is set to NULL in drm_sched_job_cleanup. But in one case, when it
>> enter into the amdgpu_device_gpu_recover, it already in
>> drm_sched_job_cleanup, and at this time, it will go to free job. But the
>> amdgpu_device_gpu_recover sometimes is faster. At that time, job is not
>> freed, but s_fence is already NULL.
No, that case can't happen. See here:
> drm_sched_job_cleanup(s_job);
>
> amdgpu_ring_priority_put(ring, s_job->s_priority);
> dma_fence_put(job->fence);
> amdgpu_sync_free(&job->sync);
> amdgpu_sync_free(&job->sched_sync);
> kfree(job);
The job itself is freed up directly after freeing the reference to the
s_fence.
So you are just papering over a much bigger problem here. This patch is
a clear NAK.
Regards,
Christian.
>>> When you see a job without an s_fence then that means the problem is
>>> somewhere else.
>>>
>>> Regards,
>>> Christian.
>>>
>>>> Signed-off-by: Emily Deng
>>>> ---
>>>>drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>>>>drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
>>>>2 files changed, 7 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> index e6ce949..5a8f08e 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> @@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct
>>> amdgpu_device *adev,
>>>> *
>>>> * job->base holds a reference to parent fence
>>>> */
>>>> - if (job && job->base.s_fence->parent &&
>>>> + if (job && job->base.s_fence && job->base.s_fence->parent &&
>>>>dma_fence_is_signaled(job->base.s_fence->parent))
>>>>job_signaled = true;
>>>>
>>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>>> index 31809ca..56cc10e 100644
>>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>>> @@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct
>>> drm_sched_job
>>>> *bad)
>>>>
>>>>spin_lock(&rq->lock);
>>>>list_for_each_entry_safe(entity, tmp,
>>>> &rq->entities,
>>> list) {
>>>> - if (bad->s_fence->scheduled.context ==
>>>> - entity->fence_context) {
>>>> + if (bad->s_fence && (bad->s_fence-
>>>> scheduled.context ==
>>>> + entity->fence_context)) {
>>>>if (atomic_read(&bad->karma) >
>>>>bad->sched->hang_limit)
>>>>if (entity->guilty)
>>>> @@ -376,7 +376,7 @@ void drm_sched_stop(struct drm_gpu_scheduler
>>> *sched, struct drm_sched_job *bad)
>>>> * This iteration is thread safe as sched thread is stopped.
>>>> */
>>>>list_for_each_entry_saf
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Ping.
Best wishes
Emily Deng
>-Original Message-
>From: amd-gfx On Behalf Of Deng,
>Emily
>Sent: Friday, November 8, 2019 10:56 AM
>To: Koenig, Christian ; amd-
>g...@lists.freedesktop.org
>Subject: RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>>-Original Message-
>>From: Christian König
>>Sent: Thursday, November 7, 2019 7:28 PM
>>To: Deng, Emily ; amd-gfx@lists.freedesktop.org
>>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>>
>>Am 07.11.19 um 11:25 schrieb Emily Deng:
>>> When the job is already signaled, the s_fence is freed. Then it will
>>> has null pointer in amdgpu_device_gpu_recover.
>>
>>NAK, the s_fence is only set to NULL when the job is destroyed. See
>>drm_sched_job_cleanup().
>I know it is set to NULL in drm_sched_job_cleanup. But in one case, when it
>enter into the amdgpu_device_gpu_recover, it already in
>drm_sched_job_cleanup, and at this time, it will go to free job. But the
>amdgpu_device_gpu_recover sometimes is faster. At that time, job is not
>freed, but s_fence is already NULL.
>>
>>When you see a job without an s_fence then that means the problem is
>>somewhere else.
>>
>>Regards,
>>Christian.
>>
>>>
>>> Signed-off-by: Emily Deng
>>> ---
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>>> drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
>>> 2 files changed, 7 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> index e6ce949..5a8f08e 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> @@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct
>>amdgpu_device *adev,
>>> *
>>> * job->base holds a reference to parent fence
>>> */
>>> - if (job && job->base.s_fence->parent &&
>>> + if (job && job->base.s_fence && job->base.s_fence->parent &&
>>> dma_fence_is_signaled(job->base.s_fence->parent))
>>> job_signaled = true;
>>>
>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>>> b/drivers/gpu/drm/scheduler/sched_main.c
>>> index 31809ca..56cc10e 100644
>>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>>> @@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct
>>drm_sched_job
>>> *bad)
>>>
>>> spin_lock(&rq->lock);
>>> list_for_each_entry_safe(entity, tmp, &rq->entities,
>>list) {
>>> - if (bad->s_fence->scheduled.context ==
>>> - entity->fence_context) {
>>> + if (bad->s_fence && (bad->s_fence-
>>>scheduled.context ==
>>> + entity->fence_context)) {
>>> if (atomic_read(&bad->karma) >
>>> bad->sched->hang_limit)
>>> if (entity->guilty)
>>> @@ -376,7 +376,7 @@ void drm_sched_stop(struct drm_gpu_scheduler
>>*sched, struct drm_sched_job *bad)
>>> * This iteration is thread safe as sched thread is stopped.
>>> */
>>> list_for_each_entry_safe_reverse(s_job, tmp, &sched-
>>>ring_mirror_list, node) {
>>> - if (s_job->s_fence->parent &&
>>> + if (s_job->s_fence && s_job->s_fence->parent &&
>>> dma_fence_remove_callback(s_job->s_fence->parent,
>>> &s_job->cb)) {
>>> atomic_dec(&sched->hw_rq_count); @@ -395,7
>+395,8 @@ void
>>> drm_sched_stop(struct drm_gpu_scheduler
>>*sched, struct drm_sched_job *bad)
>>> *
>>> * Job is still alive so fence refcount at least 1
>>> */
>>> - dma_fence_wait(&s_job->s_fence->finished, false);
>>> + if (s_job->s_fence)
>>> + dma_fence_wait(&s_job->s_fence->finished,
>>false);
>>>
>>> /*
>>> * We must keep bad job alive for later use during @@
>>-438,7
>>> +439,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool
>>full_recovery)
>>> * GPU recovers can't run in parallel.
>>> */
>>> list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list,
>>> node)
>>{
>>> - struct dma_fence *fence = s_job->s_fence->parent;
>>> + struct dma_fence *fence = s_job->s_fence ? s_job->s_fence-
>>>parent :
>>> +NULL;
>>>
>>> atomic_inc(&sched->hw_rq_count);
>>>
>
>___
>amd-gfx mailing list
>amd-gfx@lists.freedesktop.org
>https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>-Original Message-
>From: Christian König
>Sent: Thursday, November 7, 2019 7:28 PM
>To: Deng, Emily ; amd-gfx@lists.freedesktop.org
>Subject: Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
>
>Am 07.11.19 um 11:25 schrieb Emily Deng:
>> When the job is already signaled, the s_fence is freed. Then it will
>> has null pointer in amdgpu_device_gpu_recover.
>
>NAK, the s_fence is only set to NULL when the job is destroyed. See
>drm_sched_job_cleanup().
I know it is set to NULL in drm_sched_job_cleanup. But in one case, when it
enter into the amdgpu_device_gpu_recover, it already in drm_sched_job_cleanup,
and at this time, it will go to free job. But the amdgpu_device_gpu_recover
sometimes is faster. At
that time, job is not freed, but s_fence is already NULL.
>
>When you see a job without an s_fence then that means the problem is
>somewhere else.
>
>Regards,
>Christian.
>
>>
>> Signed-off-by: Emily Deng
>> ---
>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>> drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
>> 2 files changed, 7 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index e6ce949..5a8f08e 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct
>amdgpu_device *adev,
>> *
>> * job->base holds a reference to parent fence
>> */
>> -if (job && job->base.s_fence->parent &&
>> +if (job && job->base.s_fence && job->base.s_fence->parent &&
>> dma_fence_is_signaled(job->base.s_fence->parent))
>> job_signaled = true;
>>
>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>> b/drivers/gpu/drm/scheduler/sched_main.c
>> index 31809ca..56cc10e 100644
>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>> @@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct
>drm_sched_job
>> *bad)
>>
>> spin_lock(&rq->lock);
>> list_for_each_entry_safe(entity, tmp, &rq->entities,
>list) {
>> -if (bad->s_fence->scheduled.context ==
>> -entity->fence_context) {
>> +if (bad->s_fence && (bad->s_fence-
>>scheduled.context ==
>> +entity->fence_context)) {
>> if (atomic_read(&bad->karma) >
>> bad->sched->hang_limit)
>> if (entity->guilty)
>> @@ -376,7 +376,7 @@ void drm_sched_stop(struct drm_gpu_scheduler
>*sched, struct drm_sched_job *bad)
>> * This iteration is thread safe as sched thread is stopped.
>> */
>> list_for_each_entry_safe_reverse(s_job, tmp, &sched-
>>ring_mirror_list, node) {
>> -if (s_job->s_fence->parent &&
>> +if (s_job->s_fence && s_job->s_fence->parent &&
>> dma_fence_remove_callback(s_job->s_fence->parent,
>>&s_job->cb)) {
>> atomic_dec(&sched->hw_rq_count);
>> @@ -395,7 +395,8 @@ void drm_sched_stop(struct drm_gpu_scheduler
>*sched, struct drm_sched_job *bad)
>> *
>> * Job is still alive so fence refcount at least 1
>> */
>> -dma_fence_wait(&s_job->s_fence->finished, false);
>> +if (s_job->s_fence)
>> +dma_fence_wait(&s_job->s_fence->finished,
>false);
>>
>> /*
>> * We must keep bad job alive for later use during @@
>-438,7
>> +439,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool
>full_recovery)
>> * GPU recovers can't run in parallel.
>> */
>> list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node)
>{
>> -struct dma_fence *fence = s_job->s_fence->parent;
>> +struct dma_fence *fence = s_job->s_fence ? s_job->s_fence-
>>parent :
>> +NULL;
>>
>> atomic_inc(&sched->hw_rq_count);
>>
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH] drm/amdgpu: Fix the null pointer issue for tdr
Am 07.11.19 um 11:25 schrieb Emily Deng:
When the job is already signaled, the s_fence is freed. Then it will has
null pointer in amdgpu_device_gpu_recover.
NAK, the s_fence is only set to NULL when the job is destroyed. See
drm_sched_job_cleanup().
When you see a job without an s_fence then that means the problem is
somewhere else.
Regards,
Christian.
Signed-off-by: Emily Deng
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index e6ce949..5a8f08e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
*
* job->base holds a reference to parent fence
*/
- if (job && job->base.s_fence->parent &&
+ if (job && job->base.s_fence && job->base.s_fence->parent &&
dma_fence_is_signaled(job->base.s_fence->parent))
job_signaled = true;
diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index 31809ca..56cc10e 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct drm_sched_job *bad)
spin_lock(&rq->lock);
list_for_each_entry_safe(entity, tmp, &rq->entities,
list) {
- if (bad->s_fence->scheduled.context ==
- entity->fence_context) {
+ if (bad->s_fence &&
(bad->s_fence->scheduled.context ==
+ entity->fence_context)) {
if (atomic_read(&bad->karma) >
bad->sched->hang_limit)
if (entity->guilty)
@@ -376,7 +376,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct
drm_sched_job *bad)
* This iteration is thread safe as sched thread is stopped.
*/
list_for_each_entry_safe_reverse(s_job, tmp, &sched->ring_mirror_list,
node) {
- if (s_job->s_fence->parent &&
+ if (s_job->s_fence && s_job->s_fence->parent &&
dma_fence_remove_callback(s_job->s_fence->parent,
&s_job->cb)) {
atomic_dec(&sched->hw_rq_count);
@@ -395,7 +395,8 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct
drm_sched_job *bad)
*
* Job is still alive so fence refcount at least 1
*/
- dma_fence_wait(&s_job->s_fence->finished, false);
+ if (s_job->s_fence)
+ dma_fence_wait(&s_job->s_fence->finished,
false);
/*
* We must keep bad job alive for later use during
@@ -438,7 +439,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool
full_recovery)
* GPU recovers can't run in parallel.
*/
list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
- struct dma_fence *fence = s_job->s_fence->parent;
+ struct dma_fence *fence = s_job->s_fence ?
s_job->s_fence->parent : NULL;
atomic_inc(&sched->hw_rq_count);
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[PATCH] drm/amdgpu: Fix the null pointer issue for tdr
When the job is already signaled, the s_fence is freed. Then it will has
null pointer in amdgpu_device_gpu_recover.
Signed-off-by: Emily Deng
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
drivers/gpu/drm/scheduler/sched_main.c | 11 ++-
2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index e6ce949..5a8f08e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -4075,7 +4075,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
*
* job->base holds a reference to parent fence
*/
- if (job && job->base.s_fence->parent &&
+ if (job && job->base.s_fence && job->base.s_fence->parent &&
dma_fence_is_signaled(job->base.s_fence->parent))
job_signaled = true;
diff --git a/drivers/gpu/drm/scheduler/sched_main.c
b/drivers/gpu/drm/scheduler/sched_main.c
index 31809ca..56cc10e 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -334,8 +334,8 @@ void drm_sched_increase_karma(struct drm_sched_job *bad)
spin_lock(&rq->lock);
list_for_each_entry_safe(entity, tmp, &rq->entities,
list) {
- if (bad->s_fence->scheduled.context ==
- entity->fence_context) {
+ if (bad->s_fence &&
(bad->s_fence->scheduled.context ==
+ entity->fence_context)) {
if (atomic_read(&bad->karma) >
bad->sched->hang_limit)
if (entity->guilty)
@@ -376,7 +376,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct
drm_sched_job *bad)
* This iteration is thread safe as sched thread is stopped.
*/
list_for_each_entry_safe_reverse(s_job, tmp, &sched->ring_mirror_list,
node) {
- if (s_job->s_fence->parent &&
+ if (s_job->s_fence && s_job->s_fence->parent &&
dma_fence_remove_callback(s_job->s_fence->parent,
&s_job->cb)) {
atomic_dec(&sched->hw_rq_count);
@@ -395,7 +395,8 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct
drm_sched_job *bad)
*
* Job is still alive so fence refcount at least 1
*/
- dma_fence_wait(&s_job->s_fence->finished, false);
+ if (s_job->s_fence)
+ dma_fence_wait(&s_job->s_fence->finished,
false);
/*
* We must keep bad job alive for later use during
@@ -438,7 +439,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool
full_recovery)
* GPU recovers can't run in parallel.
*/
list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
- struct dma_fence *fence = s_job->s_fence->parent;
+ struct dma_fence *fence = s_job->s_fence ?
s_job->s_fence->parent : NULL;
atomic_inc(&sched->hw_rq_count);
--
2.7.4
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
