subject:"\[PATCH 5\/5\] drm\/amd\/sched\: signal and free remaining fences in amd_sched_entity

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-13 Thread Christian König


Am 13.10.2017 um 16:34 schrieb Michel Dänzer:

On 12/10/17 07:11 PM, Christian König wrote:

Am 12.10.2017 um 18:49 schrieb Michel Dänzer:

On 12/10/17 01:00 PM, Michel Dänzer wrote:

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the
huge page related changes reverted. Unfortunately, it doesn't seem to
happen reliably on every piglit run.

Can you enable KASAN in your kernel,

KASAN caught something else at the beginning of piglit, see the attached
dmesg excerpt. Not sure it's related though.

amdgpu_job_free_cb+0x13d/0x160 decodes to:

amd_sched_get_job_priority at 
.../drivers/gpu/drm/amd/amdgpu/../scheduler/gpu_scheduler.h:182

static inline enum amd_sched_priority
amd_sched_get_job_priority(struct amd_sched_job *job)
{
return (job->s_entity->rq - job->sched->sched_rq); <===
}

  (inlined by) amdgpu_job_free_cb at 
.../drivers/gpu/drm/amd/amdgpu/amdgpu_job.c:107

amdgpu_ring_priority_put(job->ring, amd_sched_get_job_priority(s_job));


Sounds a lot like the code Andres added is buggy somehow. Going to take 
a look as well.



and please look up at which line number amdgpu_vm_bo_invalidate+0x88
is.

Looks like it's this line:

if (evicted && bo->tbo.resv == vm->root.base.bo->tbo.resv) {

Maybe vm or vm->root.base.bo is NULL?

Ah, of course!

We need to reserve the page directory root when we release it or 
otherwise we can run into a race with somebody else trying to evict it.


Going to send a patch in a minute,
Christian.
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-13 Thread Michel Dänzer

On 12/10/17 07:11 PM, Christian König wrote:
> Am 12.10.2017 um 18:49 schrieb Michel Dänzer:
>> On 12/10/17 01:00 PM, Michel Dänzer wrote:
>>> [0] I also got this, but I don't know yet if it's related:
>> No, that seems to be a separate issue; I can still reproduce it with the
>> huge page related changes reverted. Unfortunately, it doesn't seem to
>> happen reliably on every piglit run.
> 
> Can you enable KASAN in your kernel,

KASAN caught something else at the beginning of piglit, see the attached
dmesg excerpt. Not sure it's related though.

amdgpu_job_free_cb+0x13d/0x160 decodes to:

amd_sched_get_job_priority at 
.../drivers/gpu/drm/amd/amdgpu/../scheduler/gpu_scheduler.h:182

static inline enum amd_sched_priority
amd_sched_get_job_priority(struct amd_sched_job *job)
{
return (job->s_entity->rq - job->sched->sched_rq); <===
}

 (inlined by) amdgpu_job_free_cb at 
.../drivers/gpu/drm/amd/amdgpu/amdgpu_job.c:107

amdgpu_ring_priority_put(job->ring, amd_sched_get_job_priority(s_job));


> and please look up at which line number amdgpu_vm_bo_invalidate+0x88
> is.

Looks like it's this line:

if (evicted && bo->tbo.resv == vm->root.base.bo->tbo.resv) {

Maybe vm or vm->root.base.bo is NULL?


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
[   89.594368] 
==
[   89.594440] BUG: KASAN: use-after-free in amdgpu_job_free_cb+0x13d/0x160 
[amdgpu]
[   89.59] Read of size 8 at addr 880367cc22c0 by task kworker/8:1/142

[   89.594449] CPU: 8 PID: 142 Comm: kworker/8:1 Tainted: GW   
4.13.0-rc5+ #29
[   89.594451] Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 
TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
[   89.594516] Workqueue: events amd_sched_job_finish [amdgpu]
[   89.594517] Call Trace:
[   89.594522]  dump_stack+0xb8/0x152
[   89.594524]  ? dma_virt_map_sg+0x1fe/0x1fe
[   89.594527]  ? show_regs_print_info+0x62/0x62
[   89.594531]  print_address_description+0x6f/0x280
[   89.594533]  kasan_report+0x27a/0x370
[   89.594596]  ? amdgpu_job_free_cb+0x13d/0x160 [amdgpu]
[   89.594599]  __asan_report_load8_noabort+0x19/0x20
[   89.594662]  amdgpu_job_free_cb+0x13d/0x160 [amdgpu]
[   89.594726]  amd_sched_job_finish+0x36e/0x630 [amdgpu]
[   89.594790]  ? trace_event_raw_event_amd_sched_process_job+0x180/0x180 
[amdgpu]
[   89.594792]  ? pick_next_task_fair+0x435/0x15c0
[   89.594795]  ? pwq_dec_nr_in_flight+0x1c2/0x4d0
[   89.594797]  ? cpu_load_update_active+0x330/0x330
[   89.594800]  ? __switch_to+0x685/0xda0
[   89.594801]  ? load_balance+0x3490/0x3490
[   89.594803]  process_one_work+0x8a5/0x1a30
[   89.594805]  ? wq_worker_sleeping+0x86/0x310
[   89.594808]  ? create_worker+0x590/0x590
[   89.594810]  ? __schedule+0x83b/0x1c80
[   89.594813]  ? schedule+0x10e/0x450
[   89.594815]  ? __schedule+0x1c80/0x1c80
[   89.594817]  ? alloc_worker+0x360/0x360
[   89.594819]  ? update_stack_state+0x402/0x780
[   89.594820]  ? update_stack_state+0x402/0x780
[   89.594822]  ? tsc_resume+0x10/0x10
[   89.594824]  worker_thread+0x21f/0x1920
[   89.594825]  ? sched_clock+0x9/0x10
[   89.594826]  ? sched_clock+0x9/0x10
[   89.594828]  ? sched_clock_local+0x43/0x130
[   89.594831]  ? process_one_work+0x1a30/0x1a30
[   89.594832]  ? pick_next_task_fair+0xcd3/0x15c0
[   89.594833]  ? cpu_load_update_active+0x330/0x330
[   89.594835]  ? __switch_to+0x685/0xda0
[   89.594836]  ? load_balance+0x3490/0x3490
[   89.594838]  ? compat_start_thread+0x80/0x80
[   89.594839]  ? sched_clock+0x9/0x10
[   89.594840]  ? sched_clock_local+0x43/0x130
[   89.594843]  ? set_rq_online.part.79+0x130/0x130
[   89.594844]  ? put_prev_entity+0x4e/0x370
[   89.594846]  ? __schedule+0x83b/0x1c80
[   89.594847]  ? kasan_kmalloc+0xad/0xe0
[   89.594849]  ? kmem_cache_alloc_trace+0xe9/0x1f0
[   89.594851]  ? firmware_map_remove+0x80/0x80
[   89.594852]  ? migrate_swap_stop+0x660/0x660
[   89.594855]  ? __schedule+0x1c80/0x1c80
[   89.594856]  ? default_wake_function+0x35/0x50
[   89.594858]  ? __wake_up_common+0xb9/0x150
[   89.594859]  ? print_dl_stats+0x80/0x80
[   89.594861]  kthread+0x310/0x3d0
[   89.594863]  ? process_one_work+0x1a30/0x1a30
[   89.594864]  ? kthread_create_on_node+0xc0/0xc0
[   89.594866]  ret_from_fork+0x25/0x30

[   89.594869] Allocated by task 1701:
[   89.594872]  save_stack_trace+0x1b/0x20
[   89.594873]  save_stack+0x43/0xd0
[   89.594874]  kasan_kmalloc+0xad/0xe0
[   89.594875]  kmem_cache_alloc_trace+0xe9/0x1f0
[   89.594913]  amdgpu_driver_open_kms+0xec/0x3f0 [amdgpu]
[   89.594925]  drm_open+0x7ea/0x13a0 [drm]
[   89.594936]  drm_stub_open+0x2a7/0x420 [drm]
[   89.594939]  chrdev_open+0x24d/0x6f0
[   89.594940]  do_dentry_open+0x5b1/0xd30
[   89.594941]  vfs_open+0xf1/0x260
[   89.594942]  path_openat+0x130a/0x5240
[   89.594944]  do_filp_open+0x23e/0x3c0
[   89.594945]

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Christian König


Am 12.10.2017 um 18:49 schrieb Michel Dänzer:

On 12/10/17 01:00 PM, Michel Dänzer wrote:

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the
huge page related changes reverted. Unfortunately, it doesn't seem to
happen reliably on every piglit run.


Can you enable KASAN in your kernel, and please look up at which line 
number amdgpu_vm_bo_invalidate+0x88 is.



Even before your changes this morning, there's another hang which
doesn't happen every time, without any corresponding dmesg output.

Lots of "fun" in amd-staging-drm-next...


Yeah, way to much stuff on my TODO list and not enough time/resources 
for extensive testing :(


Thanks for the reports,
Christian.





  BUG: unable to handle kernel NULL pointer dereference at 0220
  IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  PGD 0
  P4D 0
  
  Oops:  [#1] SMP

  Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative 
amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul 
chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic 
snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel 
drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 
i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat 
syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof 
ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd 
soundcore rng_core sg wmi parport_pc parport i2c_designware_platform 
i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid 
efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
   jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev 
hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd 
scsi_mod usbcore shpchp gpio_amdpt gpio_generic
  CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: GW  O4.13.0-rc5+ 
#28
  Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK 
(MS-7A34), BIOS 1.80 09/13/2017
  task: 9d2982c75a00 task.stack: b2744e9bc000
  RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  RSP: 0018:b2744e9bf6e8 EFLAGS: 00010202
  RAX:  RBX: 9d2848642820 RCX: 9d28c77fdae0
  RDX: 0001 RSI: 9d28c77fd800 RDI: 9d288f286008
  RBP: b2744e9bf728 R08: 00ff R09: 
  R10: 0078 R11: 9d298ba170a0 R12: 9d28c77fd800
  R13: 0001 R14: 9d288f286000 R15: 9d2848642800
  FS:  7f809fc5c300() GS:9d298e94() knlGS:
  CS:  0010 DS:  ES:  CR0: 80050033
  CR2: 0220 CR3: 00030e05a000 CR4: 003406e0
  Call Trace:
   amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
   ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
   ? ttm_bo_mem_space+0x391/0x580 [ttm]
   ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
   ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
   ttm_bo_mem_space+0x306/0x580 [ttm]
   ttm_bo_validate+0xd4/0x150 [ttm]
   ttm_bo_init_reserved+0x22e/0x440 [ttm]
   amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
   ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
   amdgpu_bo_create+0x50/0x2b0 [amdgpu]
   amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
   amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   drm_ioctl_kernel+0x5d/0xf0 [drm]
   drm_ioctl+0x32a/0x630 [drm]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   ? lru_cache_add_active_or_unevictable+0x36/0xb0
   ? __handle_mm_fault+0x90d/0xff0
   amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
   do_vfs_ioctl+0xa5/0x600
   ? handle_mm_fault+0xd8/0x230
   ? __do_page_fault+0x267/0x4c0
   SyS_ioctl+0x79/0x90
   entry_SYSCALL_64_fastpath+0x1e/0xa9
  RIP: 0033:0x7f809c8f3dc7
  RSP: 002b:7ffcc8c485f8 EFLAGS: 0246 ORIG_RAX: 0010
  RAX: ffda RBX: 7f809cbaab00 RCX: 7f809c8f3dc7
  RDX: 7ffcc8c48640 RSI: c0206440 RDI: 0006
  RBP: 4010 R08: 7f809cbaabe8 R09: 0060
  R10: 0004 R11: 0246 R12: 40001000
  R13: 7f809cbaab58 R14: 1000 R15: 7f809cbaab00
  Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 
47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 
84 24 20 02 00 00 0f 84 ab 00 00 00
  RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: b2744e9bf6e8
  CR2: 0220






___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Michel Dänzer

On 12/10/17 01:00 PM, Michel Dänzer wrote:
> 
> [0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the
huge page related changes reverted. Unfortunately, it doesn't seem to
happen reliably on every piglit run.

Even before your changes this morning, there's another hang which
doesn't happen every time, without any corresponding dmesg output.

Lots of "fun" in amd-staging-drm-next...


>  BUG: unable to handle kernel NULL pointer dereference at 0220
>  IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
>  PGD 0 
>  P4D 0 
>  
>  Oops:  [#1] SMP
>  Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative 
> amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul 
> chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic 
> snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel 
> drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 
> i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat 
> syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof 
> ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd 
> soundcore rng_core sg wmi parport_pc parport i2c_designware_platform 
> i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 
> hwmon_vid efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
>   jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod 
> evdev hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata 
> xhci_hcd scsi_mod usbcore shpchp gpio_amdpt gpio_generic
>  CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: GW  O
> 4.13.0-rc5+ #28
>  Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK 
> (MS-7A34), BIOS 1.80 09/13/2017
>  task: 9d2982c75a00 task.stack: b2744e9bc000
>  RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
>  RSP: 0018:b2744e9bf6e8 EFLAGS: 00010202
>  RAX:  RBX: 9d2848642820 RCX: 9d28c77fdae0
>  RDX: 0001 RSI: 9d28c77fd800 RDI: 9d288f286008
>  RBP: b2744e9bf728 R08: 00ff R09: 
>  R10: 0078 R11: 9d298ba170a0 R12: 9d28c77fd800
>  R13: 0001 R14: 9d288f286000 R15: 9d2848642800
>  FS:  7f809fc5c300() GS:9d298e94() knlGS:
>  CS:  0010 DS:  ES:  CR0: 80050033
>  CR2: 0220 CR3: 00030e05a000 CR4: 003406e0
>  Call Trace:
>   amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
>   ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
>   ? ttm_bo_mem_space+0x391/0x580 [ttm]
>   ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
>   ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
>   ttm_bo_mem_space+0x306/0x580 [ttm]
>   ttm_bo_validate+0xd4/0x150 [ttm]
>   ttm_bo_init_reserved+0x22e/0x440 [ttm]
>   amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
>   ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
>   amdgpu_bo_create+0x50/0x2b0 [amdgpu]
>   amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
>   amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
>   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
>   drm_ioctl_kernel+0x5d/0xf0 [drm]
>   drm_ioctl+0x32a/0x630 [drm]
>   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
>   ? lru_cache_add_active_or_unevictable+0x36/0xb0
>   ? __handle_mm_fault+0x90d/0xff0
>   amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
>   do_vfs_ioctl+0xa5/0x600
>   ? handle_mm_fault+0xd8/0x230
>   ? __do_page_fault+0x267/0x4c0
>   SyS_ioctl+0x79/0x90
>   entry_SYSCALL_64_fastpath+0x1e/0xa9
>  RIP: 0033:0x7f809c8f3dc7
>  RSP: 002b:7ffcc8c485f8 EFLAGS: 0246 ORIG_RAX: 0010
>  RAX: ffda RBX: 7f809cbaab00 RCX: 7f809c8f3dc7
>  RDX: 7ffcc8c48640 RSI: c0206440 RDI: 0006
>  RBP: 4010 R08: 7f809cbaabe8 R09: 0060
>  R10: 0004 R11: 0246 R12: 40001000
>  R13: 7f809cbaab58 R14: 1000 R15: 7f809cbaab00
>  Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 
> ed 41 c6 47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 
> 00 00 49 39 84 24 20 02 00 00 0f 84 ab 00 00 00 
>  RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: b2744e9bf6e8
>  CR2: 0220
> 
> 


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Michel Dänzer

On 12/10/17 03:50 PM, Christian König wrote:
> Am 12.10.2017 um 15:42 schrieb Michel Dänzer:
>> On 12/10/17 01:44 PM, Christian König wrote:
>>> Am 12.10.2017 um 13:00 schrieb Michel Dänzer:
>>>
 Anyway, unless anyone knows which commits from amd-staging-drm-next are
 needed to make 1d00402b4da2 stable in 4.14, the safe course of action
 seems to be reverting it (and ac7afe6b3cf3, which depends on it)?
>>> The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix
>>> placement flags in amdgpu_ttm_bind".
>> Indeed, that fixes it for me.
>>
>>> But I've assumed they went both into 4.14.
>> Unfortunately, it looks like only 1d00402b4da2 made it into 4.14. Alex,
>> please send a fixes pull for 4.14 with a backport of 70a9c6b.
>>
>> For the other issue, do we want to backport Nicolai's commits
>> 6b37d03280a4..318d85de9c20 or revert 6af0883ed977?
>>
>> Christian, can you check that there are no other fixes missing from 4.14?
> 
> Not that I know of, and manually checking is nearly impossible since I
> don't know what Alex pushed to 4.14 and what not.
> 
> Manually checking what went in and what not would take quite some time.

Looking at commits which are in drm-next-4.15 / amd-staging-drm-next but
not in Linus' tree should be a good start. Something like:

gitk v4.14-rc4..origin/amd-staging-drm-next drivers/gpu/drm/{ttm,amd}
(in an internal tree)

gitk v4.14-rc4../drm-next-4.15 drivers/gpu/drm/{ttm,amd}


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Michel Dänzer

On 12/10/17 01:44 PM, Christian König wrote:
> Am 12.10.2017 um 13:00 schrieb Michel Dänzer:
> 
>> Anyway, unless anyone knows which commits from amd-staging-drm-next are
>> needed to make 1d00402b4da2 stable in 4.14, the safe course of action
>> seems to be reverting it (and ac7afe6b3cf3, which depends on it)?
> 
> The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix
> placement flags in amdgpu_ttm_bind".

Indeed, that fixes it for me.

> But I've assumed they went both into 4.14.

Unfortunately, it looks like only 1d00402b4da2 made it into 4.14. Alex,
please send a fixes pull for 4.14 with a backport of 70a9c6b.

For the other issue, do we want to backport Nicolai's commits
6b37d03280a4..318d85de9c20 or revert 6af0883ed977?

Christian, can you check that there are no other fixes missing from 4.14?

BTW, this raises an issue: Since we push both fixes and new development
work to the same internal branch, sometimes it isn't clear which changes
should go upstream via -fixes or -next. Any ideas for mitigating the
risk of missing an important fix?

-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Christian König


Am 12.10.2017 um 13:00 schrieb Michel Dänzer:

On 12/10/17 10:05 AM, Christian König wrote:
[SNIP]

Is amd-staging-drm-next stable for you?

It seemed stable before the changes you pushed this morning. :) As of
cfb6dee86711 "drm/ttm: add transparent huge page support for cached
allocations v2", I get a flood of

  [TTM] Erroneous page count. Leaking pages.

in dmesg while running piglit, and it eventually hangs[0].


Great, going to take a look.


Anyway, unless anyone knows which commits from amd-staging-drm-next are
needed to make 1d00402b4da2 stable in 4.14, the safe course of action
seems to be reverting it (and ac7afe6b3cf3, which depends on it)?


The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix 
placement flags in amdgpu_ttm_bind". But I've assumed they went both 
into 4.14.


Thanks for the bugreport,
Christian.




[0] I also got this, but I don't know yet if it's related:

  BUG: unable to handle kernel NULL pointer dereference at 0220
  IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  PGD 0
  P4D 0
  
  Oops:  [#1] SMP

  Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative 
amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul 
chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic 
snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel 
drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 
i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat 
syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof 
ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd 
soundcore rng_core sg wmi parport_pc parport i2c_designware_platform 
i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid 
efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
   jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev 
hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd 
scsi_mod usbcore shpchp gpio_amdpt gpio_generic
  CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: GW  O4.13.0-rc5+ 
#28
  Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK 
(MS-7A34), BIOS 1.80 09/13/2017
  task: 9d2982c75a00 task.stack: b2744e9bc000
  RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  RSP: 0018:b2744e9bf6e8 EFLAGS: 00010202
  RAX:  RBX: 9d2848642820 RCX: 9d28c77fdae0
  RDX: 0001 RSI: 9d28c77fd800 RDI: 9d288f286008
  RBP: b2744e9bf728 R08: 00ff R09: 
  R10: 0078 R11: 9d298ba170a0 R12: 9d28c77fd800
  R13: 0001 R14: 9d288f286000 R15: 9d2848642800
  FS:  7f809fc5c300() GS:9d298e94() knlGS:
  CS:  0010 DS:  ES:  CR0: 80050033
  CR2: 0220 CR3: 00030e05a000 CR4: 003406e0
  Call Trace:
   amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
   ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
   ? ttm_bo_mem_space+0x391/0x580 [ttm]
   ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
   ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
   ttm_bo_mem_space+0x306/0x580 [ttm]
   ttm_bo_validate+0xd4/0x150 [ttm]
   ttm_bo_init_reserved+0x22e/0x440 [ttm]
   amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
   ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
   amdgpu_bo_create+0x50/0x2b0 [amdgpu]
   amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
   amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   drm_ioctl_kernel+0x5d/0xf0 [drm]
   drm_ioctl+0x32a/0x630 [drm]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   ? lru_cache_add_active_or_unevictable+0x36/0xb0
   ? __handle_mm_fault+0x90d/0xff0
   amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
   do_vfs_ioctl+0xa5/0x600
   ? handle_mm_fault+0xd8/0x230
   ? __do_page_fault+0x267/0x4c0
   SyS_ioctl+0x79/0x90
   entry_SYSCALL_64_fastpath+0x1e/0xa9
  RIP: 0033:0x7f809c8f3dc7
  RSP: 002b:7ffcc8c485f8 EFLAGS: 0246 ORIG_RAX: 0010
  RAX: ffda RBX: 7f809cbaab00 RCX: 7f809c8f3dc7
  RDX: 7ffcc8c48640 RSI: c0206440 RDI: 0006
  RBP: 4010 R08: 7f809cbaabe8 R09: 0060
  R10: 0004 R11: 0246 R12: 40001000
  R13: 7f809cbaab58 R14: 1000 R15: 7f809cbaab00
  Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 
47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 
84 24 20 02 00 00 0f 84 ab 00 00 00
  RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: b2744e9bf6e8
  CR2: 0220




___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Michel Dänzer

On 12/10/17 10:05 AM, Christian König wrote:
> Am 11.10.2017 um 18:30 schrieb Michel Dänzer:
>> On 28/09/17 04:55 PM, Nicolai Hähnle wrote:
>>> From: Nicolai Hähnle 
>>>
>>> Highly concurrent Piglit runs can trigger a race condition where a
>>> pending
>>> SDMA job on a buffer object is never executed because the corresponding
>>> process is killed (perhaps due to a crash). Since the job's fences were
>>> never signaled, the buffer object was effectively leaked. Worse, the
>>> buffer was stuck wherever it happened to be at the time, possibly in
>>> VRAM.
>>>
>>> The symptom was user space processes stuck in interruptible waits with
>>> kernel stacks like:
>>>
>>>  [] dma_fence_default_wait+0x112/0x250
>>>  [] dma_fence_wait_timeout+0x39/0xf0
>>>  []
>>> reservation_object_wait_timeout_rcu+0x1c2/0x300
>>>  [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
>>> [ttm]
>>>  [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
>>>  [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
>>>  [] ttm_bo_validate+0xd4/0x150 [ttm]
>>>  [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
>>>  [] amdgpu_bo_create_restricted+0x1f3/0x470
>>> [amdgpu]
>>>  [] amdgpu_bo_create+0xda/0x220 [amdgpu]
>>>  [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
>>>  [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
>>>  [] drm_ioctl+0x1fa/0x480 [drm]
>>>  [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
>>>  [] do_vfs_ioctl+0xa3/0x5f0
>>>  [] SyS_ioctl+0x79/0x90
>>>  [] entry_SYSCALL_64_fastpath+0x1e/0xad
>>>  [] 0x
>>>
>>> Signed-off-by: Nicolai Hähnle 
>>> Acked-by: Christian König 
>> Since Christian's commit which introduced the problem (6af0883ed977
>> "drm/amdgpu: discard commands of killed processes") is in 4.14, we need
>> a solution for that. Should we backport Nicolai's five commits fixing
>> the problem, or revert 6af0883ed977?

BTW, any preference for this Christian or Nicolai?

>> While looking into this, I noticed that the following commits by
>> Christian in 4.14 each also cause hangs for me when running the piglit
>> gpu profile on Tonga:
>>
>> 457e0fee04b0 "drm/amdgpu: remove the GART copy hack"
>> 1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"
>>
>> Are there fixes for these that can be backported to 4.14, or do they
>> need to be reverted there?
> Well I'm not aware that any of those two can cause problems.
> 
> For "drm/amdgpu: remove the GART copy hack" I also don't have the
> slightest idea how that could be an issue. It just removes an unused
> code path.

I also thought it's weird, and indeed I can no longer reproduce a hang
with only 457e0fee04b0; but I still can with only 1d00402b4da2. I guess
one of my bisections went wrong and incorrectly identified 457e0fee04b0
instead of 1d00402b4da2.

> Is amd-staging-drm-next stable for you?

It seemed stable before the changes you pushed this morning. :) As of
cfb6dee86711 "drm/ttm: add transparent huge page support for cached
allocations v2", I get a flood of

 [TTM] Erroneous page count. Leaking pages.

in dmesg while running piglit, and it eventually hangs[0].

Anyway, unless anyone knows which commits from amd-staging-drm-next are
needed to make 1d00402b4da2 stable in 4.14, the safe course of action
seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

[0] I also got this, but I don't know yet if it's related:

 BUG: unable to handle kernel NULL pointer dereference at 0220
 IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 PGD 0 
 P4D 0 

 Oops:  [#1] SMP
 Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative 
amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul 
chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic 
snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel 
drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 
i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat 
syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof 
ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd 
soundcore rng_core sg wmi parport_pc parport i2c_designware_platform 
i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid 
efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
  jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev 
hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd 
scsi_mod usbcore shpchp gpio_amdpt gpio_generic
 CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: GW  O4.13.0-rc5+ 
#28
 Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK 
(MS-7A34), BIOS 1.80 09/13/2017
 task: 9d2982c75a00 task.stack: b2744e9bc000
 RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 RSP: 0018:b2744e9bf6e8 EFLAGS: 00010202
 RAX:  RBX: 9d2848642820

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-12 Thread Christian König


Am 11.10.2017 um 18:30 schrieb Michel Dänzer:

On 28/09/17 04:55 PM, Nicolai Hähnle wrote:

From: Nicolai Hähnle 

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

 [] dma_fence_default_wait+0x112/0x250
 [] dma_fence_wait_timeout+0x39/0xf0
 [] reservation_object_wait_timeout_rcu+0x1c2/0x300
 [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
 [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
 [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
 [] ttm_bo_validate+0xd4/0x150 [ttm]
 [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
 [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
 [] amdgpu_bo_create+0xda/0x220 [amdgpu]
 [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
 [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
 [] drm_ioctl+0x1fa/0x480 [drm]
 [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
 [] do_vfs_ioctl+0xa3/0x5f0
 [] SyS_ioctl+0x79/0x90
 [] entry_SYSCALL_64_fastpath+0x1e/0xad
 [] 0x

Signed-off-by: Nicolai Hähnle 
Acked-by: Christian König 

Since Christian's commit which introduced the problem (6af0883ed977
"drm/amdgpu: discard commands of killed processes") is in 4.14, we need
a solution for that. Should we backport Nicolai's five commits fixing
the problem, or revert 6af0883ed977?


While looking into this, I noticed that the following commits by
Christian in 4.14 each also cause hangs for me when running the piglit
gpu profile on Tonga:

457e0fee04b0 "drm/amdgpu: remove the GART copy hack"
1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"

Are there fixes for these that can be backported to 4.14, or do they
need to be reverted there?

Well I'm not aware that any of those two can cause problems.

For "drm/amdgpu: remove the GART copy hack" I also don't have the 
slightest idea how that could be an issue. It just removes an unused 
code path.


Is amd-staging-drm-next stable for you?

Thanks,
Christian.
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-11 Thread Michel Dänzer

On 28/09/17 04:55 PM, Nicolai Hähnle wrote:
> From: Nicolai Hähnle 
> 
> Highly concurrent Piglit runs can trigger a race condition where a pending
> SDMA job on a buffer object is never executed because the corresponding
> process is killed (perhaps due to a crash). Since the job's fences were
> never signaled, the buffer object was effectively leaked. Worse, the
> buffer was stuck wherever it happened to be at the time, possibly in VRAM.
> 
> The symptom was user space processes stuck in interruptible waits with
> kernel stacks like:
> 
> [] dma_fence_default_wait+0x112/0x250
> [] dma_fence_wait_timeout+0x39/0xf0
> [] reservation_object_wait_timeout_rcu+0x1c2/0x300
> [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
> [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
> [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
> [] ttm_bo_validate+0xd4/0x150 [ttm]
> [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
> [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
> [] amdgpu_bo_create+0xda/0x220 [amdgpu]
> [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
> [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
> [] drm_ioctl+0x1fa/0x480 [drm]
> [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
> [] do_vfs_ioctl+0xa3/0x5f0
> [] SyS_ioctl+0x79/0x90
> [] entry_SYSCALL_64_fastpath+0x1e/0xad
> [] 0x
> 
> Signed-off-by: Nicolai Hähnle 
> Acked-by: Christian König 

Since Christian's commit which introduced the problem (6af0883ed977
"drm/amdgpu: discard commands of killed processes") is in 4.14, we need
a solution for that. Should we backport Nicolai's five commits fixing
the problem, or revert 6af0883ed977?


While looking into this, I noticed that the following commits by
Christian in 4.14 each also cause hangs for me when running the piglit
gpu profile on Tonga:

457e0fee04b0 "drm/amdgpu: remove the GART copy hack"
1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"

Are there fixes for these that can be backported to 4.14, or do they
need to be reverted there?


-- 
Earthling Michel Dänzer   |   http://www.amd.com
Libre software enthusiast | Mesa and X developer
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Liu, Monk

What about below plan?

1, ECANCELED is used for kernel to return an error upon guilty context 
submitting jobs, In order to prevent guilty context from submitting anymore ... 
or prevent all contexts from submitting if VRAM lost happens 
2, ENODEV is used for kernel to return error in cs_wait/fences_wait IOCTL, so 
UMD can be notified that the fence they are waiting on will never accomplished 
but just fake signaled due to GPU hang
  If you guys are picky on ENODEV, maybe ETIME/EBADFD  ??

3, ENODEV is used for kernel to return error in amdgpu_info_ioctl so UMD can 
call info_ioctl to acknowledge that the GPU is not work anymore, and after GPU 
reset done, info_ioctl can work as expected 
  Use info_ioctl there is because it doesn't need context as input parameter, 
because for VK UMD sometimes it want to query GPU healthy but no context on 
hand, only with FD/DEVICE


BR Monk



-Original Message-
From: Koenig, Christian 
Sent: 2017年10月9日 20:34
To: Haehnle, Nicolai <nicolai.haeh...@amd.com>; Liu, Monk <monk@amd.com>; 
Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org; Olsak, 
Marek <marek.ol...@amd.com>
Cc: Li, Bingley <bingley...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:
> On 09.10.2017 13:12, Christian König wrote:
>>>
>>>> Nicolai, how hard would it be to handle ENODEV as failure for all 
>>>> currently existing contexts?
>>>
>>> Impossible? "All currently existing contexts" is not a well-defined 
>>> concept when multiple drivers co-exist in the same process.
>>
>> Ok, let me refine the question: I assume there are resources "shared" 
>> between contexts like binary shader code for example which needs to 
>> be reuploaded when VRAM is lost.
>>
>> How hard would it be to handle that correctly?
>
> Okay, that makes more sense :)
>
> With the current interface it's still pretty difficult, but if we 
> could get a new per-device query ioctl which returns a "VRAM loss 
> counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. 
E.g. you could have an application which uploads shaders, a GPU reset happens 
and VRAM is lost and then the application creates a new context and makes 
submission with broken shader binaries.

So I would still vote for a separate IOCTL to reset the VRAM lost state which 
is called *before" user space starts to reupload shader/descriptors etc...

This way you also catch the case when another reset happens while you reupload 
things.

>
>>> And what would be the purpose of this? If it's to support VRAM loss, 
>>> having a per-context VRAM loss counter would enable each context to 
>>> signal ECANCELED separately.
>>
>> I thought of that on top of the -ENODEV handling.
>>
>> In other words when we see -ENODEV we call an IOCTL to let the kernel 
>> know we noticed that something is wrong and then reinit all shared 
>> resources in userspace.
>>
>> All existing context will still see -ECANCELED when we drop their 
>> command submission, but new contexts would at least not cause a new 
>> lockup immediately because their shader binaries are corrupted.
>
> I don't think we need -ENODEV for this. We just need -ECANCELED to be 
> returned when a submission is rejected due to reset (hang or VRAM loss).
>
> Mesa would keep a shadow of the VRAM loss counter in pipe_screen and 
> pipe_context, and query the kernel's counter when it encounters 
> -ECANCELED. Each context would then know to drop the CS it's built up 
> so far and restart based on comparing the VRAM loss counter of 
> pipe_screen to that of pipe_context, and similarly we could keep a 
> copy of the VRAM loss counter for important buffer objects like shader 
> binaries, descriptors, etc.
>
> This seems more robust to me than relying only on an ENODEV. We'd most 
> likely keep some kind of VRAM loss counter in Mesa *anyway* (we don't 
> maintain a list of all shaders, for example, and we can't nuke 
> important per-context across threads), and synthesizing such a counter 
> from ENODEVs is not particularly robust (what if multiple ENODEVs 
> occur for the same loss event?).
>
> BTW, I still don't like ENODEV. It seems more like the kind of error 
> code you'd return with hot-pluggable GPUs where the device can 
> physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still 
somehow note the fatality of loosing VRAM to userspace.

How about ENODATA or EBADFD?

Regards,
Christian.

>
> Cheers,
> Nicolai
>
>
>>
>> Rega

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Olsak, Marek

Mesa does not handle -ECANCELED. It only returns -ECANCELED from the Mesa 
winsys layer if the CS ioctl wasn't called (because the context is already lost 
and so the winsys doesn't submit further CS ioctls).


When the CS ioctl fails for the first time, the kernel error is returned and 
the context is marked as "lost".

The next command submission is automatically dropped by the winsys and it 
returns -ECANCELED.


Marek


From: Haehnle, Nicolai
Sent: Monday, October 9, 2017 2:58:02 PM
To: Koenig, Christian; Liu, Monk; Nicolai Hähnle; 
amd-gfx@lists.freedesktop.org; Olsak, Marek
Cc: Li, Bingley
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

On 09.10.2017 14:33, Christian König wrote:
> Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:
>> On 09.10.2017 13:12, Christian König wrote:
>>>>
>>>>> Nicolai, how hard would it be to handle ENODEV as failure for all
>>>>> currently existing contexts?
>>>>
>>>> Impossible? "All currently existing contexts" is not a well-defined
>>>> concept when multiple drivers co-exist in the same process.
>>>
>>> Ok, let me refine the question: I assume there are resources "shared"
>>> between contexts like binary shader code for example which needs to
>>> be reuploaded when VRAM is lost.
>>>
>>> How hard would it be to handle that correctly?
>>
>> Okay, that makes more sense :)
>>
>> With the current interface it's still pretty difficult, but if we
>> could get a new per-device query ioctl which returns a "VRAM loss
>> counter", it would be reasonably straight-forward.
>
> The problem with the VRAM lost counter is that this isn't save either.
> E.g. you could have an application which uploads shaders, a GPU reset
> happens and VRAM is lost and then the application creates a new context
> and makes submission with broken shader binaries.

Hmm. Here's how I imagined we'd be using a VRAM lost counter:

int si_shader_binary_upload(...)
{
...
shader->bo_vram_lost_counter = sscreen->vram_lost_counter;
shader->bo = pipe_buffer_create(...);
ptr = sscreen->b.ws->buffer_map(shader->bo->buf, ...);
... copies ...
sscreen->b.ws->buffer_unmap(shader->bo->buf);
}

int si_shader_select(...)
{
...
r = si_shader_select_with_key(ctx->sscreen, state, ...);
if (r) return r;

if (state->current->bo_vram_lost_counter !=
ctx->sscreen->vram_lost_counter) {
   ... re-upload sequence ...
}
}

(Not shown: logic that compares ctx->vram_lost_counter with
sscreen->vram_lost_counter and forces a re-validation of all state
including shaders.)

That should cover this scenario, shouldn't it?

Oh... I see one problem. But it should be easy to fix: when creating a
new amdgpu context, Mesa needs to query the vram lost counter. So then
the sequence of events would be either:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Mesa creates a new amdgpu context, queries the VRAM lost counter --> 1
- si_screen::vram_lost_counter is updated to 1
- Draw happens on the new context --> si_shader_select will catch the
VRAM loss

Or:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Mesa creates a new amdgpu context, VRAM lost counter still 0
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Draw happens on the new context and proceeds normally
...
- Mesa flushes the CS, and the kernel will return an error code because
the device VRAM lost counter is different from the amdgpu context VRAM
lost counter


> So I would still vote for a separate IOCTL to reset the VRAM lost state
> which is called *before" user space starts to reupload
> shader/descriptors etc...

The question is: is that separate IOCTL per-context or per-fd? If it's
per-fd, then it's not compatible with multiple drivers. If it's
per-context, then I don't see how it helps. Perhaps you could explain?


 > This way you also catch the case when another reset happens while you
 > re-upload things.

My assumption would be that the re-upload happens *after* the new amdgpu
context is created. Then the repeat reset should be caught by the kernel
when we try to submit a CS on the new context (this is assuming that the
kernel context's vram lost counter is initialized properly when the
context is created):

- Mesa prepares upload, sets shader->bo_vram_lost_counter to 0
- Mesa uploads a shader binary
- While doing this, a GPU reset happens[0], kernel increments device
VRAM lost counter to 1
- Draw happens with the new shader, Mesa proceeds normally
...
- On flush / CS submit, the kernel detects the VR

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle

It depends on what you mean by "handle". If amdgpu_cs_submit_raw were to 
return ECANCELED, the correct error message would be printed.

We don't do any of the "trying to continue" business because back when 
we last discussed that we said that it wasn't such a great idea, and to 
be honest, it really isn't a great idea for normal applications. For the 
X server / compositor it could be valuable though.

Cheers,
Nicolai

On 09.10.2017 15:57, Olsak, Marek wrote:
Mesa does not handle -ECANCELED. It only returns -ECANCELED from the 
Mesa winsys layer if the CS ioctl wasn't called (because the context is 
already lost and so the winsys doesn't submit further CS ioctls).

When the CS ioctl fails for the first time, the kernel error is returned 
and the context is marked as "lost".

The next command submission is automatically dropped by the winsys and 
it returns -ECANCELED.

Marek

*From:* Haehnle, Nicolai
*Sent:* Monday, October 9, 2017 2:58:02 PM
*To:* Koenig, Christian; Liu, Monk; Nicolai Hähnle; 
amd-gfx@lists.freedesktop.org; Olsak, Marek

*Cc:* Li, Bingley
*Subject:* Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini

On 09.10.2017 14:33, Christian König wrote:

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

On 09.10.2017 13:12, Christian König wrote:

Nicolai, how hard would it be to handle ENODEV as failure for all 
currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined 
concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" 
between contexts like binary shader code for example which needs to 
be reuploaded when VRAM is lost.

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we 
could get a new per-device query ioctl which returns a "VRAM loss 
counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. 
E.g. you could have an application which uploads shaders, a GPU reset 
happens and VRAM is lost and then the application creates a new context 
and makes submission with broken shader binaries.

Hmm. Here's how I imagined we'd be using a VRAM lost counter:

int si_shader_binary_upload(...)
{
     ...
     shader->bo_vram_lost_counter = sscreen->vram_lost_counter;
     shader->bo = pipe_buffer_create(...);
     ptr = sscreen->b.ws->buffer_map(shader->bo->buf, ...);
     ... copies ...
     sscreen->b.ws->buffer_unmap(shader->bo->buf);
}

int si_shader_select(...)
{
     ...
     r = si_shader_select_with_key(ctx->sscreen, state, ...);
     if (r) return r;

     if (state->current->bo_vram_lost_counter !=
     ctx->sscreen->vram_lost_counter) {
    ... re-upload sequence ...
     }
}

(Not shown: logic that compares ctx->vram_lost_counter with
sscreen->vram_lost_counter and forces a re-validation of all state
including shaders.)

That should cover this scenario, shouldn't it?

Oh... I see one problem. But it should be easy to fix: when creating a
new amdgpu context, Mesa needs to query the vram lost counter. So then
the sequence of events would be either:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Mesa creates a new amdgpu context, queries the VRAM lost counter --> 1
- si_screen::vram_lost_counter is updated to 1
- Draw happens on the new context --> si_shader_select will catch the
VRAM loss

Or:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Mesa creates a new amdgpu context, VRAM lost counter still 0
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Draw happens on the new context and proceeds normally
...
- Mesa flushes the CS, and the kernel will return an error code because
the device VRAM lost counter is different from the amdgpu context VRAM
lost counter

So I would still vote for a separate IOCTL to reset the VRAM lost state 
which is called *before" user space starts to reupload 
shader/descriptors etc...

The question is: is that separate IOCTL per-context or per-fd? If it's
per-fd, then it's not compatible with multiple drivers. If it's
per-context, then I don't see how it helps. Perhaps you could explain?

  > This way you also catch the case when another reset happens while you
  > re-upload things.

My assumption would be that the re-upload happens *after* the new amdgpu
context is created. Then the repeat reset should be caught by the kernel
when we try to submit a CS on the new context (this is assuming that the
kernel context's vram lost counter is initialized properly when the
context is created):

- Mesa prepar

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle

.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)

founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error 
on fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)

founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang 
hit



BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0

[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
   [] ttm_bo_validate+0xd4/0x150 [ttm]
   [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
   [] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
   [] amdgpu_bo_create+0xda/0x220 [amdgpu]
   [] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
   [] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
   [] drm_ioctl+0x1fa/0x480 [drm]
   [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
   [] do_vfs_ioctl+0xa3/0x5f0
   [] SyS_ioctl+0x79/0x90
   [] entry_SYSCALL_64_fastpath+0x1e/0xad
   [] 0x

Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
Acked-by: Christian König <christian.koe...@amd.com>
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
    1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
amd_gpu_scheduler *sched,
amd_sched_entity_is_idle(entity));
    amd_sched_rq_remove_entity(rq, entity);
    if (r) {
    struct amd_sched_job *job;
    /* Park the kernel for a moment to make sure it isn't
processing
 * our enity.
 */
    kthread_park(sched->thread);
    kthread_unpark(sched->thread);
-    while (kfifo_out(>job_queue, , sizeof(job)))
+    while (kfifo_out(>job_queue, , sizeof(job))) {
+    struct amd_sched_fence *s_fence = job->s_fence;
+    amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on
s_fence->finished before it is signaled, use 
dma_fence_set_error() for this.


Additional to that it would be nice to note in the subject line that
this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König
<christian.koe...@amd.com>.

Regards,
Christian.


+ amd_sched_fence_finished(s_fence);
+    dma_fence_put(_fence->finished);
    sched->ops->free_job(job);
+    }
    }
    kfifo_free(>job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct
dma_fence_cb *cb)
    {
    struct amd_sched_entity *entity =
    container_of(

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle

e drivers co-exist in the same process.


And what would be the purpose of this? If it's to support VRAM loss, 
having a per-context VRAM loss counter would enable each context to 
signal ECANCELED separately.


Cheers,
Nicolai




Monk, would it be ok with you when we return ENODEV only for 
existing context when VRAM is lost and/or we have a strict mode GPU 
reset? E.g. newly created contexts would continue work as they should.


Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a 
consequence, Mesa would already handle a kernel error of ECANCELED 
on context loss correctly :)


Cheers,
Nicolai

On 09.10.2017 12:35, Liu, Monk wrote:

Hi Christian

You reject some of my patches that returns -ENODEV, with the 
cause that MESA doesn't do the handling on -ENODEV


But if Nicolai can confirm that MESA do have a handling on 
-ECANCELED, then we need to overall align our error code, on 
detail below IOCTL can return error code:


Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is 
signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset 
happened after the process created (because for strict mode we 
block process instead of context)



according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly 
speaking, kernel part doesn't have any place with "-ECANCELED" so 
this solution on MESA side doesn't align with *current* amdgpu 
driver,
which only return 0 on success or -EINVALID on other error but 
definitely no "-ECANCELED" error code,


so if we talking about community rules we shouldn't let MESA 
handle -ECANCELED ,  we should have a unified error code


+ Marek

BR Monk




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk 
<monk....@amd.com>; Nicolai Hähnle <nhaeh...@gmail.com>; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:
For gpu reset patches (already submitted to pub) I would make 
kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)
founded as error, that way UMD would run into robust extension 
path

and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error 
on fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use 
dma_fence_set_error()

for this.
For gpu reset patches (already submitted to pub) I would make 
kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)
founded as error, that way UMD would run into robust extension 
path

and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process 
being
killed or crashed like Nicolai hit ... since there is no gpu 
hang hit



BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition 
where a

pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). 
Since the
job's fences were never signaled, the buffer object was 
effectively

leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0

[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
   [] ttm_bo_validate+0xd4/0x150 [ttm]
   [] ttm_bo_init_reserved+0x2ed/0x420 
[ttm]
   [] 
amdgpu_bo_create_restricted+0x1f3/0x470

[amdgpu]
   [] amdgpu_bo_create+0xda/0x220

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Christian König

s instead of context)



according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly 
speaking, kernel part doesn't have any place with "-ECANCELED" so 
this solution on MESA side doesn't align with *current* amdgpu 
driver,
which only return 0 on success or -EINVALID on other error but 
definitely no "-ECANCELED" error code,


so if we talking about community rules we shouldn't let MESA 
handle -ECANCELED ,  we should have a unified error code


+ Marek

BR Monk




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk 
<monk@amd.com>; Nicolai Hähnle <nhaeh...@gmail.com>; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:
For gpu reset patches (already submitted to pub) I would make 
kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)
founded as error, that way UMD would run into robust extension 
path

and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error 
on fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use 
dma_fence_set_error()

for this.
For gpu reset patches (already submitted to pub) I would make 
kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)
founded as error, that way UMD would run into robust extension 
path

and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process 
being
killed or crashed like Nicolai hit ... since there is no gpu 
hang hit



BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition 
where a

pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). 
Since the
job's fences were never signaled, the buffer object was 
effectively

leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0

[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
   [] ttm_bo_validate+0xd4/0x150 [ttm]
   [] ttm_bo_init_reserved+0x2ed/0x420 
[ttm]
   [] 
amdgpu_bo_create_restricted+0x1f3/0x470

[amdgpu]
   [] amdgpu_bo_create+0xda/0x220 [amdgpu]
   [] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
   [] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
   [] drm_ioctl+0x1fa/0x480 [drm]
   [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
   [] do_vfs_ioctl+0xa3/0x5f0
   [] SyS_ioctl+0x79/0x90
   [] entry_SYSCALL_64_fastpath+0x1e/0xad
   [] 0x

Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
Acked-by: Christian König <christian.koe...@amd.com>
---
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
amd_gpu_scheduler *sched,
amd_sched_entity_is_idle(entity));
amd_sched_rq_remove_entity(rq, entity);
if (r) {
struct amd_sched_job *job;
/* Park the kernel for a moment to make sure it isn't
processing
 * our enity.
 */
kthread_park(sched->thread);
kthread_unpark(sched->thread);
-while (kfifo_out(>job_queue, , sizeof(job)))
+while (kfifo_out(>job_queue, , 
sizeof(job))) {

+struct amd_sched_fence *s_fence = job->s_fen

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle


On 09.10.2017 12:59, Christian König wrote:
Nicolai, how hard would it be to handle ENODEV as failure for all 
currently existing contexts?


Impossible? "All currently existing contexts" is not a well-defined 
concept when multiple drivers co-exist in the same process.


And what would be the purpose of this? If it's to support VRAM loss, 
having a per-context VRAM loss counter would enable each context to 
signal ECANCELED separately.


Cheers,
Nicolai




Monk, would it be ok with you when we return ENODEV only for existing 
context when VRAM is lost and/or we have a strict mode GPU reset? E.g. 
newly created contexts would continue work as they should.


Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a 
consequence, Mesa would already handle a kernel error of ECANCELED on 
context loss correctly :)


Cheers,
Nicolai

On 09.10.2017 12:35, Liu, Monk wrote:

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause 
that MESA doesn't do the handling on -ENODEV


But if Nicolai can confirm that MESA do have a handling on 
-ECANCELED, then we need to overall align our error code, on detail 
below IOCTL can return error code:


Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is 
signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset 
happened after the process created (because for strict mode we block 
process instead of context)



according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, 
kernel part doesn't have any place with "-ECANCELED" so this solution 
on MESA side doesn't align with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but 
definitely no "-ECANCELED" error code,


so if we talking about community rules we shouldn't let MESA handle 
-ECANCELED ,  we should have a unified error code


+ Marek

BR Monk




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk 
<monk@amd.com>; Nicolai Hähnle <nhaeh...@gmail.com>; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error on 
fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang hit


BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Christian König



Nicolai, how hard would it be to handle ENODEV as failure for all 
currently existing contexts?


Impossible? "All currently existing contexts" is not a well-defined 
concept when multiple drivers co-exist in the same process.


Ok, let me refine the question: I assume there are resources "shared" 
between contexts like binary shader code for example which needs to be 
reuploaded when VRAM is lost.


How hard would it be to handle that correctly?

And what would be the purpose of this? If it's to support VRAM loss, 
having a per-context VRAM loss counter would enable each context to 
signal ECANCELED separately.


I thought of that on top of the -ENODEV handling.

In other words when we see -ENODEV we call an IOCTL to let the kernel 
know we noticed that something is wrong and then reinit all shared 
resources in userspace.


All existing context will still see -ECANCELED when we drop their 
command submission, but new contexts would at least not cause a new 
lockup immediately because their shader binaries are corrupted.


Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

On 09.10.2017 12:59, Christian König wrote:
Nicolai, how hard would it be to handle ENODEV as failure for all 
currently existing contexts?


Impossible? "All currently existing contexts" is not a well-defined 
concept when multiple drivers co-exist in the same process.


And what would be the purpose of this? If it's to support VRAM loss, 
having a per-context VRAM loss counter would enable each context to 
signal ECANCELED separately.


Cheers,
Nicolai




Monk, would it be ok with you when we return ENODEV only for existing 
context when VRAM is lost and/or we have a strict mode GPU reset? 
E.g. newly created contexts would continue work as they should.


Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a 
consequence, Mesa would already handle a kernel error of ECANCELED 
on context loss correctly :)


Cheers,
Nicolai

On 09.10.2017 12:35, Liu, Monk wrote:

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause 
that MESA doesn't do the handling on -ENODEV


But if Nicolai can confirm that MESA do have a handling on 
-ECANCELED, then we need to overall align our error code, on detail 
below IOCTL can return error code:


Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is 
signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset 
happened after the process created (because for strict mode we 
block process instead of context)



according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly 
speaking, kernel part doesn't have any place with "-ECANCELED" so 
this solution on MESA side doesn't align with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but 
definitely no "-ECANCELED" error code,


so if we talking about community rules we shouldn't let MESA handle 
-ECANCELED ,  we should have a unified error code


+ Marek

BR Monk




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk 
<monk@amd.com>; Nicolai Hähnle <nhaeh...@gmail.com>; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)

founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error 
on fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences 
IOCTL)

founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang 
hit



BR Monk




-Original Message-
From: amd-gfx [mailto:a

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Christian König


Hi Nicolai & Monk,

well that sounds like exactly what I wanted to hear.

When Mesa already handles ECANCELED gracefully as return code for a lost 
context then I think we can move forward with that. Monk, would that be 
ok with you?


Nicolai, how hard would it be to handle ENODEV as failure for all 
currently existing contexts?


Monk, would it be ok with you when we return ENODEV only for existing 
context when VRAM is lost and/or we have a strict mode GPU reset? E.g. 
newly created contexts would continue work as they should.


Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a 
consequence, Mesa would already handle a kernel error of ECANCELED on 
context loss correctly :)


Cheers,
Nicolai

On 09.10.2017 12:35, Liu, Monk wrote:

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause 
that MESA doesn't do the handling on -ENODEV


But if Nicolai can confirm that MESA do have a handling on 
-ECANCELED, then we need to overall align our error code, on detail 
below IOCTL can return error code:


Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is 
signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset 
happened after the process created (because for strict mode we block 
process instead of context)



according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, 
kernel part doesn't have any place with "-ECANCELED" so this solution 
on MESA side doesn't align with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but 
definitely no "-ECANCELED" error code,


so if we talking about community rules we shouldn't let MESA handle 
-ECANCELED ,  we should have a unified error code


+ Marek

BR Monk




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk 
<monk@amd.com>; Nicolai Hähnle <nhaeh...@gmail.com>; 
amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


On 09.10.2017 10:02, Christian König wrote:

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV 
upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and 
treats those as context lost. Perhaps we could use the same error on 
fences?

That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang hit


BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
   [] ttm_bo_validate+0xd4/0x150 [ttm]
   [] ttm_bo_init_reserved+0x2ed/0x420

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle


Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a 
consequence, Mesa would already handle a kernel error of ECANCELED on 
context loss correctly :)


Cheers,
Nicolai

On 09.10.2017 12:35, Liu, Monk wrote:

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA 
doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we 
need to overall align our error code, on detail below IOCTL can return error 
code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with 
error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after 
the process created (because for strict mode we block process instead of 
context)


according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't 
have any place with "-ECANCELED" so this solution on MESA side doesn't align 
with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but definitely no 
"-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED 
,  we should have a unified error code

+ Marek

BR Monk

  




-Original Message-
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk <monk@amd.com>; 
Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

On 09.10.2017 10:02, Christian König wrote:

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Well that is only closed source behavior which is completely
irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those 
as context lost. Perhaps we could use the same error on fences?
That makes more sense to me than -ENODEV.

Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on
s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang hit


BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On
Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>;
amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining
fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at
the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits
with kernel stacks like:

   [] dma_fence_default_wait+0x112/0x250
   [] dma_fence_wait_timeout+0x39/0xf0
   []
reservation_object_wait_timeout_rcu+0x1c2/0x300
   [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
   [] ttm_bo_validate+0xd4/0x150 [ttm]
   [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
   [] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
   [] amdgpu_bo_create+0xda/0x220 [amdgpu]
   [] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
   [] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
   [] drm_ioctl+0x1fa/0x480 [drm]
   [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
   [] do_vfs_ioctl+0xa3/0x5f0
   [] SyS_ioctl+0x79/0x90
   [] entry_SYSCALL_64_fastpath+0x1e/0xad
   [] 0x

Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
Acked-by: Christian König <christian.koe...@amd.com>
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Liu, Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA 
doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we 
need to overall align our error code, on detail below IOCTL can return error 
code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl


My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with 
error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after 
the process created (because for strict mode we block process instead of 
context)


according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel 
part doesn't have any place with "-ECANCELED" so this solution on MESA side 
doesn't align with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but definitely no 
"-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED 
,  we should have a unified error code 

+ Marek

BR Monk

 



-Original Message-
From: Haehnle, Nicolai 
Sent: 2017年10月9日 18:14
To: Koenig, Christian <christian.koe...@amd.com>; Liu, Monk <monk@amd.com>; 
Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

On 09.10.2017 10:02, Christian König wrote:
>> For gpu reset patches (already submitted to pub) I would make kernel 
>> return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) 
>> founded as error, that way UMD would run into robust extension path 
>> and considering the GPU hang occurred,
> Well that is only closed source behavior which is completely 
> irrelevant for upstream development.
> 
> As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those 
as context lost. Perhaps we could use the same error on fences? 
That makes more sense to me than -ENODEV.

Cheers,
Nicolai

> 
> Regards,
> Christian.
> 
> Am 09.10.2017 um 08:42 schrieb Liu, Monk:
>> Christian
>>
>>> It would be really nice to have an error code set on 
>>> s_fence->finished before it is signaled, use dma_fence_set_error() 
>>> for this.
>> For gpu reset patches (already submitted to pub) I would make kernel 
>> return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) 
>> founded as error, that way UMD would run into robust extension path 
>> and considering the GPU hang occurred,
>>
>> Don't know if this is expected for the case of normal process being 
>> killed or crashed like Nicolai hit ... since there is no gpu hang hit
>>
>>
>> BR Monk
>>
>>
>>
>>
>> -Original Message-
>> From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On 
>> Behalf Of Christian K?nig
>> Sent: 2017年9月28日 23:01
>> To: Nicolai Hähnle <nhaeh...@gmail.com>; 
>> amd-gfx@lists.freedesktop.org
>> Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
>> Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
>> fences in amd_sched_entity_fini
>>
>> Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:
>>> From: Nicolai Hähnle <nicolai.haeh...@amd.com>
>>>
>>> Highly concurrent Piglit runs can trigger a race condition where a 
>>> pending SDMA job on a buffer object is never executed because the 
>>> corresponding process is killed (perhaps due to a crash). Since the 
>>> job's fences were never signaled, the buffer object was effectively 
>>> leaked. Worse, the buffer was stuck wherever it happened to be at 
>>> the time, possibly in VRAM.
>>>
>>> The symptom was user space processes stuck in interruptible waits 
>>> with kernel stacks like:
>>>
>>>   [] dma_fence_default_wait+0x112/0x250
>>>   [] dma_fence_wait_timeout+0x39/0xf0
>>>   []
>>> reservation_object_wait_timeout_rcu+0x1c2/0x300
>>>   [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
>>> [ttm]
>>>   [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
>>>   [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
>>>   [] ttm_bo_validate+0xd4/0x150 [ttm]
>>>   [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
>>>   [] amdgpu_bo_create_restricted+0x1f3/0x470
>>> [amdgpu]
>>>   [] amdgpu_bo_create+0xda/0x220 [amdgpu]
>>>   [] amdgpu_gem_object_create+0xaa/0x140 
>>>

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Nicolai Hähnle


On 09.10.2017 10:02, Christian König wrote:
For gpu reset patches (already submitted to pub) I would make kernel 
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) 
founded as error, that way UMD would run into robust extension path 
and considering the GPU hang occurred,
Well that is only closed source behavior which is completely irrelevant 
for upstream development.


As far as I know we haven't pushed the change to return -ENODEV upstream.


FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats 
those as context lost. Perhaps we could use the same error on fences? 
That makes more sense to me than -ENODEV.


Cheers,
Nicolai



Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian

It would be really nice to have an error code set on 
s_fence->finished before it is signaled, use dma_fence_set_error() 
for this.
For gpu reset patches (already submitted to pub) I would make kernel 
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) 
founded as error, that way UMD would run into robust extension path 
and considering the GPU hang occurred,


Don't know if this is expected for the case of normal process being 
killed or crashed like Nicolai hit ... since there is no gpu hang hit



BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On Behalf 
Of Christian K?nig

Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining 
fences in amd_sched_entity_fini


Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the 
time, possibly in VRAM.


The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

  [] dma_fence_default_wait+0x112/0x250
  [] dma_fence_wait_timeout+0x39/0xf0
  [] 
reservation_object_wait_timeout_rcu+0x1c2/0x300
  [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 
[ttm]

  [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
  [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
  [] ttm_bo_validate+0xd4/0x150 [ttm]
  [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
  [] amdgpu_bo_create_restricted+0x1f3/0x470 
[amdgpu]

  [] amdgpu_bo_create+0xda/0x220 [amdgpu]
  [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
  [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
  [] drm_ioctl+0x1fa/0x480 [drm]
  [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
  [] do_vfs_ioctl+0xa3/0x5f0
  [] SyS_ioctl+0x79/0x90
  [] entry_SYSCALL_64_fastpath+0x1e/0xad
  [] 0x

Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
Acked-by: Christian König <christian.koe...@amd.com>
---
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
   1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct 
amd_gpu_scheduler *sched,

   amd_sched_entity_is_idle(entity));
   amd_sched_rq_remove_entity(rq, entity);
   if (r) {
   struct amd_sched_job *job;
   /* Park the kernel for a moment to make sure it isn't 
processing

    * our enity.
    */
   kthread_park(sched->thread);
   kthread_unpark(sched->thread);
-    while (kfifo_out(>job_queue, , sizeof(job)))
+    while (kfifo_out(>job_queue, , sizeof(job))) {
+    struct amd_sched_fence *s_fence = job->s_fence;
+    amd_sched_fence_scheduled(s_fence);
It would be really nice to have an error code set on s_fence->finished 
before it is signaled, use dma_fence_set_error() for this.


Additional to that it would be nice to note in the subject line that 
this is a rather important bug fix.


With that fixed the whole series is Reviewed-by: Christian König 
<christian.koe...@amd.com>.


Regards,
Christian.


+    amd_sched_fence_finished(s_fence);
+    dma_fence_put(_fence->finished);
   sched->ops->free_job(job);
+    }
   }
   kfifo_free(>job_queue);
   }
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct 
dma_fence_cb *cb)

   {
   struct amd_sched_entity *entity =
   container_of(cb, struct amd_sched_entity, cb);
   entity->depe

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Christian König


For gpu reset patches (already submitted to pub) I would make kernel return 
-ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as 
error, that way UMD would run into robust extension path and considering the 
GPU hang occurred,
Well that is only closed source behavior which is completely irrelevant 
for upstream development.


As far as I know we haven't pushed the change to return -ENODEV upstream.

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Christian


It would be really nice to have an error code set on s_fence->finished before 
it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return 
-ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as 
error, that way UMD would run into robust extension path and considering the 
GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or 
crashed like Nicolai hit ... since there is no gpu hang hit


BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On Behalf Of 
Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle <nicolai.haeh...@amd.com>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the
corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, 
possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

  [] dma_fence_default_wait+0x112/0x250
  [] dma_fence_wait_timeout+0x39/0xf0
  [] reservation_object_wait_timeout_rcu+0x1c2/0x300
  [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
  [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
  [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
  [] ttm_bo_validate+0xd4/0x150 [ttm]
  [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
  [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
  [] amdgpu_bo_create+0xda/0x220 [amdgpu]
  [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
  [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
  [] drm_ioctl+0x1fa/0x480 [drm]
  [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
  [] do_vfs_ioctl+0xa3/0x5f0
  [] SyS_ioctl+0x79/0x90
  [] entry_SYSCALL_64_fastpath+0x1e/0xad
  [] 0x

Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
Acked-by: Christian König <christian.koe...@amd.com>
---
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
   1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
*sched,
amd_sched_entity_is_idle(entity));
amd_sched_rq_remove_entity(rq, entity);
if (r) {
struct amd_sched_job *job;
   
   		/* Park the kernel for a moment to make sure it isn't processing

 * our enity.
 */
kthread_park(sched->thread);
kthread_unpark(sched->thread);
-   while (kfifo_out(>job_queue, , sizeof(job)))
+   while (kfifo_out(>job_queue, , sizeof(job))) {
+   struct amd_sched_fence *s_fence = job->s_fence;
+   amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before 
it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a 
rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König 
<christian.koe...@amd.com>.

Regards,
Christian.


+   amd_sched_fence_finished(s_fence);
+   dma_fence_put(_fence->finished);
sched->ops->free_job(job);
+   }
   
   	}

kfifo_free(>job_queue);
   }
   
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)

   {
struct amd_sched_entity *entity =
container_of(cb, struct amd_sched_entity, cb);
entity->dependency = NULL;


___
amd-gfx mailing list
amd-gfx@list

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-09 Thread Liu, Monk

Christian

> It would be really nice to have an error code set on s_fence->finished before 
> it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return 
-ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as 
error, that way UMD would run into robust extension path and considering the 
GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or 
crashed like Nicolai hit ... since there is no gpu hang hit 


BR Monk




-Original Message-
From: amd-gfx [mailto:amd-gfx-boun...@lists.freedesktop.org] On Behalf Of 
Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <nhaeh...@gmail.com>; amd-gfx@lists.freedesktop.org
Cc: Haehnle, Nicolai <nicolai.haeh...@amd.com>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in 
amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:
> From: Nicolai Hähnle <nicolai.haeh...@amd.com>
>
> Highly concurrent Piglit runs can trigger a race condition where a 
> pending SDMA job on a buffer object is never executed because the 
> corresponding process is killed (perhaps due to a crash). Since the 
> job's fences were never signaled, the buffer object was effectively 
> leaked. Worse, the buffer was stuck wherever it happened to be at the time, 
> possibly in VRAM.
>
> The symptom was user space processes stuck in interruptible waits with 
> kernel stacks like:
>
>  [] dma_fence_default_wait+0x112/0x250
>  [] dma_fence_wait_timeout+0x39/0xf0
>  [] reservation_object_wait_timeout_rcu+0x1c2/0x300
>  [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
>  [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
>  [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
>  [] ttm_bo_validate+0xd4/0x150 [ttm]
>  [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
>  [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
>  [] amdgpu_bo_create+0xda/0x220 [amdgpu]
>  [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
>  [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
>  [] drm_ioctl+0x1fa/0x480 [drm]
>  [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
>  [] do_vfs_ioctl+0xa3/0x5f0
>  [] SyS_ioctl+0x79/0x90
>  [] entry_SYSCALL_64_fastpath+0x1e/0xad
>  [] 0x
>
> Signed-off-by: Nicolai Hähnle <nicolai.haeh...@amd.com>
> Acked-by: Christian König <christian.koe...@amd.com>
> ---
>   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
> b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> index 54eb77cffd9b..32a99e980d78 100644
> --- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> +++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> @@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
> *sched,
>   amd_sched_entity_is_idle(entity));
>   amd_sched_rq_remove_entity(rq, entity);
>   if (r) {
>   struct amd_sched_job *job;
>   
>   /* Park the kernel for a moment to make sure it isn't processing
>* our enity.
>*/
>   kthread_park(sched->thread);
>   kthread_unpark(sched->thread);
> - while (kfifo_out(>job_queue, , sizeof(job)))
> + while (kfifo_out(>job_queue, , sizeof(job))) {
> + struct amd_sched_fence *s_fence = job->s_fence;
> + amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before 
it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a 
rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König 
<christian.koe...@amd.com>.

Regards,
Christian.

> + amd_sched_fence_finished(s_fence);
> + dma_fence_put(_fence->finished);
>   sched->ops->free_job(job);
> + }
>   
>   }
>   kfifo_free(>job_queue);
>   }
>   
>   static void amd_sched_entity_wakeup(struct dma_fence *f, struct 
> dma_fence_cb *cb)
>   {
>   struct amd_sched_entity *entity =
>   container_of(cb, struct amd_sched_entity, cb);
>   entity->dependency = NULL;


___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-10-02 Thread Tom St Denis


Hi Nicolai,

Ping?  :-)

Tom


On 28/09/17 11:01 AM, Christian König wrote:

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle 

Highly concurrent Piglit runs can trigger a race condition where a 
pending

SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in 
VRAM.


The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

 [] dma_fence_default_wait+0x112/0x250
 [] dma_fence_wait_timeout+0x39/0xf0
 [] reservation_object_wait_timeout_rcu+0x1c2/0x300
 [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
 [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
 [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
 [] ttm_bo_validate+0xd4/0x150 [ttm]
 [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
 [] amdgpu_bo_create_restricted+0x1f3/0x470 
[amdgpu]

 [] amdgpu_bo_create+0xda/0x220 [amdgpu]
 [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
 [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
 [] drm_ioctl+0x1fa/0x480 [drm]
 [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
 [] do_vfs_ioctl+0xa3/0x5f0
 [] SyS_ioctl+0x79/0x90
 [] entry_SYSCALL_64_fastpath+0x1e/0xad
 [] 0x

Signed-off-by: Nicolai Hähnle 
Acked-by: Christian König 
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
  1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct 
amd_gpu_scheduler *sched,

  amd_sched_entity_is_idle(entity));
  amd_sched_rq_remove_entity(rq, entity);
  if (r) {
  struct amd_sched_job *job;
  /* Park the kernel for a moment to make sure it isn't 
processing

   * our enity.
   */
  kthread_park(sched->thread);
  kthread_unpark(sched->thread);
-    while (kfifo_out(>job_queue, , sizeof(job)))
+    while (kfifo_out(>job_queue, , sizeof(job))) {
+    struct amd_sched_fence *s_fence = job->s_fence;
+    amd_sched_fence_scheduled(s_fence);


It would be really nice to have an error code set on s_fence->finished 
before it is signaled, use dma_fence_set_error() for this.


Additional to that it would be nice to note in the subject line that 
this is a rather important bug fix.


With that fixed the whole series is Reviewed-by: Christian König 
.


Regards,
Christian.


+    amd_sched_fence_finished(s_fence);
+    dma_fence_put(_fence->finished);
  sched->ops->free_job(job);
+    }
  }
  kfifo_free(>job_queue);
  }
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct 
dma_fence_cb *cb)

  {
  struct amd_sched_entity *entity =
  container_of(cb, struct amd_sched_entity, cb);
  entity->dependency = NULL;



___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx


___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-09-28 Thread Chunming Zhou




On 2017年09月28日 22:55, Nicolai Hähnle wrote:

From: Nicolai Hähnle 

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in VRAM.

Indeed good catch.

Cheers,
David Zhou


The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

 [] dma_fence_default_wait+0x112/0x250
 [] dma_fence_wait_timeout+0x39/0xf0
 [] reservation_object_wait_timeout_rcu+0x1c2/0x300
 [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
 [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
 [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
 [] ttm_bo_validate+0xd4/0x150 [ttm]
 [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
 [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
 [] amdgpu_bo_create+0xda/0x220 [amdgpu]
 [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
 [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
 [] drm_ioctl+0x1fa/0x480 [drm]
 [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
 [] do_vfs_ioctl+0xa3/0x5f0
 [] SyS_ioctl+0x79/0x90
 [] entry_SYSCALL_64_fastpath+0x1e/0xad
 [] 0x

Signed-off-by: Nicolai Hähnle 
Acked-by: Christian König 
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
  1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
*sched,
amd_sched_entity_is_idle(entity));
amd_sched_rq_remove_entity(rq, entity);
if (r) {
struct amd_sched_job *job;
  
  		/* Park the kernel for a moment to make sure it isn't processing

 * our enity.
 */
kthread_park(sched->thread);
kthread_unpark(sched->thread);
-   while (kfifo_out(>job_queue, , sizeof(job)))
+   while (kfifo_out(>job_queue, , sizeof(job))) {
+   struct amd_sched_fence *s_fence = job->s_fence;
+   amd_sched_fence_scheduled(s_fence);
+   amd_sched_fence_finished(s_fence);
+   dma_fence_put(_fence->finished);
sched->ops->free_job(job);
+   }
  
  	}

kfifo_free(>job_queue);
  }
  
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)

  {
struct amd_sched_entity *entity =
container_of(cb, struct amd_sched_entity, cb);
entity->dependency = NULL;


___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-09-28 Thread Marek Olšák

Thanks for this series. I can finally finish piglit on VI.

Marek

On Thu, Sep 28, 2017 at 4:55 PM, Nicolai Hähnle  wrote:
> From: Nicolai Hähnle 
>
> Highly concurrent Piglit runs can trigger a race condition where a pending
> SDMA job on a buffer object is never executed because the corresponding
> process is killed (perhaps due to a crash). Since the job's fences were
> never signaled, the buffer object was effectively leaked. Worse, the
> buffer was stuck wherever it happened to be at the time, possibly in VRAM.
>
> The symptom was user space processes stuck in interruptible waits with
> kernel stacks like:
>
> [] dma_fence_default_wait+0x112/0x250
> [] dma_fence_wait_timeout+0x39/0xf0
> [] reservation_object_wait_timeout_rcu+0x1c2/0x300
> [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
> [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
> [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
> [] ttm_bo_validate+0xd4/0x150 [ttm]
> [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
> [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
> [] amdgpu_bo_create+0xda/0x220 [amdgpu]
> [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
> [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
> [] drm_ioctl+0x1fa/0x480 [drm]
> [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
> [] do_vfs_ioctl+0xa3/0x5f0
> [] SyS_ioctl+0x79/0x90
> [] entry_SYSCALL_64_fastpath+0x1e/0xad
> [] 0x
>
> Signed-off-by: Nicolai Hähnle 
> Acked-by: Christian König 
> ---
>  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
> b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> index 54eb77cffd9b..32a99e980d78 100644
> --- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> +++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
> @@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
> *sched,
> amd_sched_entity_is_idle(entity));
> amd_sched_rq_remove_entity(rq, entity);
> if (r) {
> struct amd_sched_job *job;
>
> /* Park the kernel for a moment to make sure it isn't 
> processing
>  * our enity.
>  */
> kthread_park(sched->thread);
> kthread_unpark(sched->thread);
> -   while (kfifo_out(>job_queue, , sizeof(job)))
> +   while (kfifo_out(>job_queue, , sizeof(job))) {
> +   struct amd_sched_fence *s_fence = job->s_fence;
> +   amd_sched_fence_scheduled(s_fence);
> +   amd_sched_fence_finished(s_fence);
> +   dma_fence_put(_fence->finished);
> sched->ops->free_job(job);
> +   }
>
> }
> kfifo_free(>job_queue);
>  }
>
>  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb 
> *cb)
>  {
> struct amd_sched_entity *entity =
> container_of(cb, struct amd_sched_entity, cb);
> entity->dependency = NULL;
> --
> 2.11.0
>
> ___
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-09-28 Thread Christian König


Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

From: Nicolai Hähnle 

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

 [] dma_fence_default_wait+0x112/0x250
 [] dma_fence_wait_timeout+0x39/0xf0
 [] reservation_object_wait_timeout_rcu+0x1c2/0x300
 [] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
 [] ttm_mem_evict_first+0xba/0x1a0 [ttm]
 [] ttm_bo_mem_space+0x341/0x4c0 [ttm]
 [] ttm_bo_validate+0xd4/0x150 [ttm]
 [] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
 [] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
 [] amdgpu_bo_create+0xda/0x220 [amdgpu]
 [] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
 [] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
 [] drm_ioctl+0x1fa/0x480 [drm]
 [] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
 [] do_vfs_ioctl+0xa3/0x5f0
 [] SyS_ioctl+0x79/0x90
 [] entry_SYSCALL_64_fastpath+0x1e/0xad
 [] 0x

Signed-off-by: Nicolai Hähnle 
Acked-by: Christian König 
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
  1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
*sched,
amd_sched_entity_is_idle(entity));
amd_sched_rq_remove_entity(rq, entity);
if (r) {
struct amd_sched_job *job;
  
  		/* Park the kernel for a moment to make sure it isn't processing

 * our enity.
 */
kthread_park(sched->thread);
kthread_unpark(sched->thread);
-   while (kfifo_out(>job_queue, , sizeof(job)))
+   while (kfifo_out(>job_queue, , sizeof(job))) {
+   struct amd_sched_fence *s_fence = job->s_fence;
+   amd_sched_fence_scheduled(s_fence);


It would be really nice to have an error code set on s_fence->finished 
before it is signaled, use dma_fence_set_error() for this.


Additional to that it would be nice to note in the subject line that 
this is a rather important bug fix.


With that fixed the whole series is Reviewed-by: Christian König 
.


Regards,
Christian.


+   amd_sched_fence_finished(s_fence);
+   dma_fence_put(_fence->finished);
sched->ops->free_job(job);
+   }
  
  	}

kfifo_free(>job_queue);
  }
  
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)

  {
struct amd_sched_entity *entity =
container_of(cb, struct amd_sched_entity, cb);
entity->dependency = NULL;



___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

[PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

2017-09-28 Thread Nicolai Hähnle

From: Nicolai Hähnle 

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding
process is killed (perhaps due to a crash). Since the job's fences were
never signaled, the buffer object was effectively leaked. Worse, the
buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with
kernel stacks like:

[] dma_fence_default_wait+0x112/0x250
[] dma_fence_wait_timeout+0x39/0xf0
[] reservation_object_wait_timeout_rcu+0x1c2/0x300
[] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
[] ttm_mem_evict_first+0xba/0x1a0 [ttm]
[] ttm_bo_mem_space+0x341/0x4c0 [ttm]
[] ttm_bo_validate+0xd4/0x150 [ttm]
[] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
[] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
[] amdgpu_bo_create+0xda/0x220 [amdgpu]
[] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
[] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
[] drm_ioctl+0x1fa/0x480 [drm]
[] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
[] do_vfs_ioctl+0xa3/0x5f0
[] SyS_ioctl+0x79/0x90
[] entry_SYSCALL_64_fastpath+0x1e/0xad
[] 0x

Signed-off-by: Nicolai Hähnle 
Acked-by: Christian König 
---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c 
b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
index 54eb77cffd9b..32a99e980d78 100644
--- a/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
+++ b/drivers/gpu/drm/amd/scheduler/gpu_scheduler.c
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler 
*sched,
amd_sched_entity_is_idle(entity));
amd_sched_rq_remove_entity(rq, entity);
if (r) {
struct amd_sched_job *job;
 
/* Park the kernel for a moment to make sure it isn't processing
 * our enity.
 */
kthread_park(sched->thread);
kthread_unpark(sched->thread);
-   while (kfifo_out(>job_queue, , sizeof(job)))
+   while (kfifo_out(>job_queue, , sizeof(job))) {
+   struct amd_sched_fence *s_fence = job->s_fence;
+   amd_sched_fence_scheduled(s_fence);
+   amd_sched_fence_finished(s_fence);
+   dma_fence_put(_fence->finished);
sched->ops->free_job(job);
+   }
 
}
kfifo_free(>job_queue);
 }
 
 static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb 
*cb)
 {
struct amd_sched_entity *entity =
container_of(cb, struct amd_sched_entity, cb);
entity->dependency = NULL;
-- 
2.11.0

___
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

[PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

29 matches

Site Navigation

Mail list logo

Footer information