[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #15 from Daniel T.  ---
Okay I've updated to 12.0.1 and managed to get a trace of Octodad: Dadliest
Catch (Was a good starter because it always crashes consistently before the
main menu)

Here's the trace
https://drive.google.com/open?id=0Bzmfxv--_ou4NHIwU1ZNeURrRnc

In qapitrace the first call that causes a lockup is Frame 90; Call 30070

Various good calls:
26916
27306
28899
29045
29249
29980
30013
30055
30065
30067
30069 <-- last good

Various bad calls:
34530
31749
30200
30089
30075
30070 <-- first bad

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #14 from iive at yahoo.com ---
@Daniel T.
FYI 12.1 would be the next release. The first bugfix release 12.0.1 is already
out (fixes compilation with debugging).
The shader backend is very specific to r600 cards and very few people even
touch it. So there is no way this bugs gets fixed by accident.

@eydee,
Are you using environment variables for the override in drirc file?
`
export MESA_GL_VERSION_OVERRIDE=4.1
export MESA_GLSL_VERSION_OVERRIDE=410
`
should set override for any programs started from this bash session onward.

If you prefer drirc override, then have in mind that `glretrace` is the program
that does the actual playback/replay.

If you upload a trace I could try to narrow it down, but my card also needs the
override.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #13 from eydee  ---
Adding some info, I have very similar GPU crashes (including kernel log) on
6850 BARTS using any recent (11.x, 12.x) Mesa version up until latest git.

The GPU lockup and crash happens in Bioshock Infinite, right after the intro
logos and before reaching the title screen. It is not random, there's no way to
start the game at all without touching Mesa configuration. However, launching
with R600_DEBUG=nosb makes the game start up perfectly. Everything works, no
visual glitches either, only performance is lower than expected, possibly
because of the lack of shader optimization.

I can do an apitrace if needed, but can't replay and analyze it, as apitrace
can't replay on forced OpenGL context. (BARTS only exposes 3.3, overriding is
required to start the game.)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #12 from Daniel T.  ---
Cheers for the thorough reply, before I try your debugging tips I will wait for
12.1(first bugfix release) to filter down to archlinux in hopes that mesa 12
fixes some of these already so I'm not wasting my time chasing bugs that are
already fixed. I'll update in a few weeks with new info. thanks

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #11 from iive at yahoo.com ---
I'd ask you to try and narrow down the issue.

Get the program `apitrace`. Set `R600_DEBUG=nosb` and use apitrace with some of
the games that cause a crash. It should produce a trace file. That trace file
should be able to reproduce the lockup when replaying(retracing) it without the
"nosb".
(Smaller traces help with the next steps. Also use any trick to limit the fps
and keep scenes simple. e.g. lock the dpm in low power mode, set vblank/vsync,
etc)

Then try to narrow down what causes the lockup. You can use `qapitrace` GUI
program for that. Load the trace, and use "lookup state" (right mouse button on
a frame or just double click) to replay to a specific frame. Find out the frame
that causes lockup.

(E.g. lookup every 100'th frame. When you find a crash, try every 10'th
starting from last known working frame. Then lookup frames one by one.
It might be less optimal than binary search, but rebooting and reloading
probably takes more time.)

After you find the frame, try to find the exact call inside that frame that
causes the lockup. You can "look up" individual calls, just like with frames,
so use the same process. For the final narrow down you can use that only draw
calls could cause a lockup, because they are the one that execute the shaders
(usually 1 vertex and 1 pixel/fragment).

At this point you can upload the archived trace file somewhere (usually traces
are quite big) and tell us the location and the number of the call that causes
the lockup.

Doing this saves a lot of time to the developers, because each lookup replays
the trace file from the begging to the selected call. As you can see, the
process is tedious and time consuming.


(It might be good idea to trim the trace file a few frames after the first
frame that crashes, just to make it smaller. Make sure you don't cut too much
and that you report the correct call number for the file you upload).
---

Optionally, you can also start `qapitrace` with `R600_DEBUG=nosb` and lookup
the crashing draw call. That would let you see the exact vertex and fragment
(pixel) shaders that cause the problem (in GLSL form).

If you have Mesa3D compiled with debug support you may try to get the
disassembled output of the shaders.

This is done by setting `R600_DEBUG=sbdry,ps,vs` and then looking up the draw
call that crashes.

The "sbdry" option allows the "shader backend" to run, but then prevents the
use the result. This allows us to print the crashing code, but not use it and
crash the system.

The "ps,vs" options would dump the disassembly of the pixel and vertex shaders,
before and after "shader backend" (the one disabled by nosb).

Since the binary shader code is compiled and optimized by SB at first use, The
last vertex and fragment(pixel) shaders are the ones that cause the hang.

These steps are optional, because developers can easily do them on their own
and they already build mesa with debug enabled.

---

There is a faster way to get the disassembly of the crashing shaders.
Set `R600_DEBUG=ps,vs` and run the game. The last shaders that are output
should be the ones that cause the crash. You just have to be sure that the real
info is not lost at the lockup. (Getting partial log could be quite
misleading.)
---
btw, I'm not developer. Just advanced user.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #10 from Adam Lyall  ---
Hi, I've been get this issue with my Radeon HD5850 as well. Grid Autosport
always crashes at the same place in its benchmark (just before the big hill)
but I found if I disable the SB Shader optimizer (R600_DEBUG=nosb) the game
runs perfectly throughout.

My GPU temperature is fine (tops out at 62c on a hot day) so that does not seem
to be an issue.

I've dumped the shaders for both when the game crashes with SB and when it is
running fine with nosb in hopes they might help (I've followed what Jürgen
Scholz did at issue 93352#c7). I won't be near this computer for a few weeks
now but if you let me know if there is more info you would like me to gather in
the next 24 hours I'll see what I can do.

FYI my system is:
Ubuntu 16.04
Mesa from padoka PPA (12.1~git160603151200.a64c7cd~x~padoka0)
SAPPHIRE VAPOR-X Radeon HD 5870
AMD Phenom II 955

For now I'm just attaching the shader dump from the crashed SB version. If you
want them I'll upload the nosb variant as well? I've also attached the script I
made for debugging the game.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #9 from Adam Lyall  ---
Created attachment 124359
  --> https://bugs.freedesktop.org/attachment.cgi?id=124359=edit
Grid Autosport crash GLSL dump SB

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

--- Comment #8 from Marcin Juszkiewicz  ---
Can you guys check gfx card temperature when it crashes? I have other GPU in
desktop now so can not check.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: 

[Bug 94900] HD6950 GPU lockup loop with various steam games (octodad, saints row 4, grid autosport)

[bugzilla-dae...@freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=94900

Daniel T.  changed:

   What|Removed |Added

Summary|HD6950 GPU lockup with  |HD6950 GPU lockup loop with
   |various steam games |various steam games
   |(octodad, saints row 4) |(octodad, saints row 4,
   ||grid autosport)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-- next part --
An HTML attachment was scrubbed...
URL: