| Issue |
241948
|
| Summary |
[AMDGPU][gfx125x] Is waiting for global_inv required when it's the only outstanding LOADcnt op?
|
| Labels |
new issue
|
| Assignees |
|
| Reporter |
danzimm
|
On gfx125x, `global_inv` increments LOADcnt, but writes no VGPRs. In [\#169647](https://github.com/llvm/llvm-project/pull/169647) this fact was used to elide `s_wait_loadcnt` instructions at function returns when `global_inv` is the only outstanding LOADcnt operation.
I think we can do the same simplification (eliding waits when the only outstanding LOADcnt operation is `global_inv`) to the interior of functions. The goal of this issue is to confirm whether that simplification is safe.
## Background
The waits in the interior currently come from [SIMemoryLegalizer](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp#L2105-L2106), where they’re inserted after `global_inv` on gfx125x. This was introduced in [\#175865](https://github.com/llvm/llvm-project/pull/175865), with a comment stating the wait is needed to "ensure that the proceeding INV has completed as it may get reorded with following load instructions." However, I couldn’t find support for that concern in the ISA guide (see the discussion of loads below).
## Viability
To ensure the optimization is viable, we need to make sure an outstanding `global_inv` won’t cause data races with other loads or stores. The references below come from the [CDNA5 ISA Guide](https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/instruction-set-architectures/amd-instinct-cdna5-instruction-set-architecture.pdf).
### Load Ordering
Let’s start with loads. From the CDNA5 docs:
- §5.7.1: "Instructions within a type return in the order they were issued."
- §5.7.1.2: "LOADcnt counts the number of issued-but-not-completed instructions of type: load, atomic-with-return, and cache-invalidate."
Consequently, loads and `global_inv` will retire in issue order. Presumably this also means the `global_inv` takes effect before any later loads (regardless of whether or not a wait comes immediately after `global_inv`).
### Store Ordering
Since stores use a different wait counter, we also need to verify they don’t an issue independently. To begin, consider a `global_inv` followed by a store. We will verify below that it is ok for the store to retire before `global_inv`, regardless of the scope of either operation (the reverse, of a `global_inv` retiring before a store is equivalent to inserting a wait prior to the store, and thus is trivially safe).
#### Breaking down global\_inv SCOPE cases
Table 13 from the ISA doc will be useful for the arguments below. It indicates how each scope bit interacts with the different levels of cache:
| Scope | WGP\$ | L2\$ |
| :---- | :---- | :---- |
| 0-WGP | NOP | NOP |
| 1-SE | wb/wbinv/inv | NOP |
| 2-DEV | wb/wbinv/inv | NOP |
| 3-SYS | wb/wbinv/inv | wb/wbinv/inv |
When the `global_inv` scope is WGP, then the instruction is a nop, so it doesn’t matter if it retires before or after the store. When the scope is SE/DEV, then data from L2\$ will come from the issued store (since L2\$ won’t be invalidated), regardless of whether the invalidate or store retires first.
All that’s left to show is that there is no race when the scope of the `global_inv` is SYS. In this situation, `global_inv` must either write dirty L2\$ lines back to memory or keep them in L2\$. To see why, consider the following instruction sequence:
```
global_store_b32 v0, v1, s[0:1] scope:SCOPE_DEV th:TH_STORE_WB ; TH_STORE_WB => L2$ dirty
s_wait_storecnt 0x0
global_inv scope:SCOPE_SYS
global_load_b32 v3, v0, s[0:1] scope:SCOPE_DEV
```
As per §5.7.1.2: "There is also no hazard between a memory store to one memory location followed by a load from the same location (if both are from the same wave) \- the load and store to the same address stay in order", we must have `v3 == v1`, but that can only be guaranteed if `global_inv` either keeps the dirty L2 line, or writes its data back to memory first.
So if, after a store, a load from L2\$ returns the stored data (regardless of the store’s scope), the claim follows: a store that retires before a `global_inv scope:SCOPE_SYS` has effectively written to L2\$, and, hence, as illustrated by the preceding example, `global_inv` retiring afterward won’t lose that data. The next section shows that the key assumption, that L2\$ returns the stored data, holds.
#### L2\$ never returns stale data after a store
>From §4.1.1:
At each level of the cache hierarchy:
- If the ISA.scope \<= Cache-scope
- Reads can hit into the cache
- Writes into the cache and transaction is acknowledged from this level of cache
- Read-modify-write operations can occur locally in this cache
- Else: must "forward" the operation to the next larger scope cache
- Forward \= load cannot hit in this cache, store must propagate to the next higher cache (but updates state of this cache level as well)
Hence, a store will write its data to either the WGP\$ (when ISA.scope \== SCOPE\_WGP), L2\$ (when the store’s ISA.scope \== SCOPE\_SE or SCOPE\_DEV), or memory (when ISA.scope \== SCOPE\_SYS). In the first case, since WGP\$ only holds clean data (§4.1.2: "The I\$, K\$ and WGP\$ hold clean data only"), the store must also write to L2\$. In the last case, L2\$ "updates state of this cache level as well". I’m not sure if this means L2\$ will get a copy of the data or the address will be evicted so that future loads come from memory; either way, a subsequent read from L2\$ will never return stale data.
### Conclusion
The above assessment is quite technical. My intuition came from the fact that if [\#169647](https://github.com/llvm/llvm-project/pull/169647) is safe, then I *think* it shouldn’t matter if the optimization is applied at a function boundary or not (I guess function boundaries are more meaningful to us than they are to the hardware/counters).
The change to simplify post-`global_inv` waits would be relatively small inside of SIInsertWaitcnts (likely just calling `simplifyVmemInvOnlyWait` somewhere in the 125x codepaths).
### AI Disclosure
I wrote every word, but used claude (a) to double check my logic (b) double check my citations/help find better citations (c) grammar/spelling check.
Cc @PankajDwivedi-25, @Pierre-vh, @rampitec, @jayfoad
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs