This is an automated email from the ASF dual-hosted git repository.
tqchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm-tirx.git
The following commit(s) were added to refs/heads/main by this push:
new f43b3030d [FEAT] Support PTX ISA 9.5 and the remaining sm_107 PTX
instructions (#241)
f43b3030d is described below
commit f43b3030d33196d3a5f21e56b37549a6d3a5c70b
Author: Hongyi Jin <[email protected]>
AuthorDate: Thu Nov 12 09:52:32 2026 -0500
[FEAT] Support PTX ISA 9.5 and the remaining sm_107 PTX instructions (#241)
This PR extends the `T.ptx` table to PTX ISA 9.5 (CUDA 13.5) and fills
the sm_107 (Rubin) gaps that remained after the PTX 9.4 update.
## PTX ISA 9.5 additions (`_PTX_95_ENTRIES`)
- `fabric.submit.commit_group` and `fabric.wait.sync_restrict::read N`
(sm_100+). The old `fabric.wait.sync_restrict::reads` stays registered
and is noted as deprecated.
- `cp.async.bulk.wait_group.sync_restrict::proxy::async N` (sm_100+).
- `fence.alias.sync_restrict::read.release.sys` (sm_90+).
- `sin`/`cos` `.2pi`, which takes its input in cycles. It is an optional
slot on the existing entries, and a check enforces the ISA rule that
`.2pi` requires `.ftz`.
- `tensormap.replace` `.alloc_class` (the `global_address` form and the
field-less form) and `tensormap.query.alloc_class_large` (sm_90a and the
sm_100f/110f/120f families).
- `%locality_domain_id` / `%locality_domain_supplemental_sm`.
`T.cuda.mov_sreg` already reads them; this adds tests.
## sm_107 gaps and fixes
- The packed-integer `add/sub{.sat}.{u8x4,s8x4}`,
`add.sat.{u32,u16x2,s16x2}`, `neg.s8x4` and `min/max{.relu}` on
`.u8x4`/`.s8x4` were excluded as "sm_120f-only". The ISA gives them to
sm_107f and sm_120f, and ptxas agrees. They are now sm_107f sibling
entries, so the existing families keep their floors.
- `fabric.try_{get,put,red}.async.tensor` (PTX 9.4, sm_107f; 25 entries
covering TMA's override shapes) and `fabric.try_atom` (PTX 9.4,
sm_100+).
- `tcgen05.shift` (sm_100a/103a/107a/110a).
- **Fix:** the PTX 9.3 `fabric.try_*` entries stored the completion
mechanism as one modifier token containing dots. `T.ptx["..."]` and
attribute chains split on dots, so these instructions could not be
called from a program at all. The mechanism is now split into separate
slots. Rendered PTX and helper names are byte-identical. A new
table-wide test rejects dotted tokens.
- The module header now states the table is certified against CUDA 13.5.
## Validation
- **ptxas:** arch floors were measured with CUDA 13.5 ptxas and recorded
in `MEASURED` notes. Where ptxas is more permissive than the ISA, the
table follows the ISA:
`cp.async.bulk.wait_group.sync_restrict::proxy::async` assembles at
sm_90, and `fabric.try_atom` accepts `.min/.max.s32`.
- **Full certification:** `PTX_CERT=1` certifies the whole table under
CUDA 13.5, with no regressions in existing entries.
- **On a VR200 (sm_107a)**, GPU tests check:
- the 9.5 waits execute;
- `sin`/`cos .2pi` meets the ISA error bounds;
- the tensor-map allocation class seen through the query after each
replace, including PTX 9.5's clarification that `global_address` without
`.alloc_class` resets the class to Small;
- the locality-domain registers;
- lane-wise wrap-around, saturation and relu semantics of all 16
packed-integer forms against numpy.
- **Assembled only, not executed:** the fabric tensor ops and `try_atom`
need host-created logical endpoints across NVLink peers, and
`tcgen05.shift` would need a full tmem round trip.
- The hard-coded variant count in `test_ptx_all_variants_render_unique`
and the address-operand census in `test_ptx_addr.py` are updated. With
the new entries removed, the census gives back the old numbers exactly.
## Notes
- **Query `.shared::cta` not registered:** CUDA 13.5 ptxas rejects
`tensormap.query.alloc_class_large.shared::cta` with a 32-bit address
register ("Arguments mismatch for instruction 'mov'"), but accepts a
64-bit register or a symbol. Our shared-address ABI uses 32-bit
registers, so that form is left out and the table notes why. It looks
like a ptxas bug.
- **Not included:** `trace.mark` and `.metadata_section`. The directive
needs module-level PTX emission, which deserves its own design.
- **Running on sm_107:** torch (cu132) pulls in a pip NVRTC 13.2 that
cannot target sm_107, and `cuda.pathfinder` prefers it. Running these
tests on sm_107 needs the CUDA 13.5 NVRTC loaded first, e.g. via
`LD_PRELOAD`. Separately,
`test_sparse_decode_conversion_intrinsics_codegen` fails on CUDA 13.5 on
`main` as well: ptxas requires `sm_100a/f` for the `cvt` forms it
compiles at plain `sm_100`.
---
python/tvm_tirx/backend/cuda/ptx/table.py | 617 ++++++++++++++++++++--
python/tvm_tirx/script/tirx.pyi | 91 +++-
tests/python/tirx/codegen/test_cuda.py | 64 +++
tests/python/tirx/codegen/test_ptx_addr.py | 8 +-
tests/python/tirx/codegen/test_ptx_dialect.py | 706 +++++++++++++++++++++++++-
5 files changed, 1391 insertions(+), 95 deletions(-)
diff --git a/python/tvm_tirx/backend/cuda/ptx/table.py
b/python/tvm_tirx/backend/cuda/ptx/table.py
index ac283dca1..e304e53fb 100644
--- a/python/tvm_tirx/backend/cuda/ptx/table.py
+++ b/python/tvm_tirx/backend/cuda/ptx/table.py
@@ -20,14 +20,19 @@ Pure data + pure functions: this module deliberately
imports nothing from
``tvm`` so the thin generators (``gen_stubs``, ``gen_coverage``,
``gen_helpers``) can load it standalone.
-Section and table numbers cite PTX ISA 9.4, the version implemented by the
-CUDA 13.4 ptxas this table is certified against:
-``https://docs.nvidia.com/cuda/developer-preview/13.4/parallel-thread-execution/index.html``.
-Every ``MEASURED`` note records CUDA 13.4 ptxas behaviour. ``_PTX_94_ENTRIES``
-groups the instructions and qualifiers PTX ISA 9.4 introduced (the SM107
-family additions) ahead of the long-standing families in ``_ENTRIES``; it is a
-grouping by ISA release, not a second document version -- every entry cites
-the same document.
+The table is certified against the CUDA 13.5 ptxas, which implements PTX ISA
+9.5: ``https://docs.nvidia.com/cuda/parallel-thread-execution/index.html``.
+Section and table numbers cite PTX ISA 9.4
+(``https://docs.nvidia.com/cuda/developer-preview/13.4/parallel-thread-execution/index.html``)
+except in ``_PTX_95_ENTRIES``, which cites 9.5; 9.5 inserts a few sections
+(e.g. 9.7.10.30 tensormap.query.alloc_class_large), shifting the numbers after
+them by one. A ``MEASURED`` note records the ptxas behaviour of the toolkit it
+was written against -- CUDA 13.4 unless it says CUDA 13.5 -- and the whole
+table, those notes included, re-certifies under CUDA 13.5 (``PTX_CERT=1``).
+``_PTX_94_ENTRIES`` and ``_PTX_95_ENTRIES`` group the instructions and
+qualifiers PTX ISA 9.4 (the SM107 family additions) and 9.5 introduced ahead of
+the long-standing families in ``_ENTRIES``; they are groupings by ISA release,
+not separate document versions to certify against.
The converged :class:`InstructionEntry` design:
@@ -1179,20 +1184,20 @@ def _check_int_addsub(m, mnemonic):
sub.type1 d, a, b; .type1 = {.u16, .u32, .u64, .s16, .s64}
sub{.sat}.type2 d, a, b; .type2 = {.s32, .u8x4, .s8x4}
- Of the saturating forms only `.s32` is registered. `add.sat` on
- .u32/.u16x2/.s16x2 arrived in PTX ISA 9.2 as "sm_120f or higher in the same
- family". The remaining types in these entries belong to syntax lines that
- never carry `.sat`: .u16/.u64/.s16/.s64 for add, and
- .u16/.u32/.u64/.s16/.s64 for sub. The `.u8x4`/`.s8x4` types are excluded
- outright because they are sm_120f-only (see `_MINMAX_INT_PLAIN` below).
- `add.sat.s32` and `sub.sat.s32` are the PTX 1.0 forms.
+ Of the saturating forms only `.s32` is in these entries: `add.sat.s32` and
+ `sub.sat.s32` are the PTX 1.0 forms. `add.sat` on .u32/.u16x2/.s16x2 and
+ the `.u8x4`/`.s8x4` types arrived in PTX ISA 9.2 for "sm_107f or higher in
+ the same family, sm_120f or higher in the same family", and are the sm_107f
+ `add_int_sm107f`/`sub_int_sm107f` entries. The remaining types here belong
+ to syntax lines that never carry `.sat`: .u16/.u64/.s16/.s64 for add, and
+ .u16/.u32/.u64/.s16/.s64 for sub.
"""
if not m["sat"] or m["type"] == "s32":
return None
if mnemonic == "add" and m["type"] in ("u32", "u16x2", "s16x2"):
return (
- f"add.sat.{m['type']} is an sm_120f-only form (PTX ISA 9.2); "
- ".sat is registered on .s32 only"
+ f"add.sat.{m['type']} is an sm_107f/sm_120f form (PTX ISA 9.2), "
+ "registered as the add_int_sm107f entry"
)
return f".sat is not on the {mnemonic}.{m['type']} syntax line"
@@ -1207,6 +1212,24 @@ def _check_int_sub(m):
return _check_int_addsub(m, "sub")
+def _check_int_add_sm107f(m):
+ """The add lines PTX gives to sm_107f/sm_120f (9.7.1.1 Target ISA Notes).
+
+ `add{.sat}.{u8x4,s8x4}` and `add.sat.{u32,u16x2,s16x2}`; the non-saturating
+ .u32/.u16x2/.s16x2 forms are the sm_90 `add_int` entry's.
+ """
+ if m["type"] in ("u32", "u16x2", "s16x2") and not m["sat"]:
+ return f"add.{m['type']} without .sat is the add_int entry (sm_90
floor)"
+ return None
+
+
+def _check_minmax_int8x4(m):
+ """`.relu` is on the signed line only: `min{.relu}.s8x4`, `min.u8x4`."""
+ if m["relu"] and m["type"] != "s8x4":
+ return ".relu applies only to the signed .s8x4 line"
+ return None
+
+
def _check_int_mad(m):
"""`.sat` on the multiply-add lines: `.hi` mode, `.s32` type, nothing else.
@@ -1221,9 +1244,9 @@ def _check_int_mad(m):
# The .type tokens of the integer min/max lines (PTX ISA 9.7.1.13/9.7.1.14).
# `.relu` is only on the signed line (type2); the unsigned/other line takes
none.
-# NOT REGISTERED: `.u8x4` and `{.relu}.s8x4`, which the ISA supports only on
-# "sm_120f or higher in the same family" -- outside the architectures this
-# dialect certifies, and ptxas rejects them at sm_100.
+# `.u8x4` and `{.relu}.s8x4` are not here: the ISA gives them only to sm_107f
+# and sm_120f (and ptxas rejects them at sm_100), so they are the sm_107f
+# `{min,max}_int8x4` sibling entries.
_MINMAX_INT_PLAIN = ("u16", "u32", "u64", "u16x2", "s16", "s64")
_MINMAX_INT_RELU = ("s16x2", "s32")
# The floating-point and half-precision lines (9.7.3.11/9.7.3.12,
9.7.4.7/9.7.4.8).
@@ -3235,6 +3258,78 @@ def _tma_global_dim_stride_operands(bracket, size_type):
)
+# Attribute-override shapes of a fabric tensor handle, as for TMA's
+# `.override_attribute` (PTX ISA "Overriding tensor property value"): no
+# override; `.override::global_dim` on 1D tensors (a singleton size vector); or
+# `.override::global_dim_stride` on 2D-5D tensors (sizes, lower strides, upper
+# stride). The size vector binds .b8 or .b16 registers, one entry each.
+_FABRIC_TENSOR_OVERRIDES = (
+ ("", None, ("1d", "2d", "3d", "4d", "5d"), None),
+ ("_global_dim_b8", "override::global_dim", ("1d",), "tma_size_b8"),
+ ("_global_dim_b16", "override::global_dim", ("1d",), "b16"),
+ (
+ "_global_dim_stride_b8",
+ "override::global_dim_stride",
+ ("2d", "3d", "4d", "5d"),
+ "tma_size_b8",
+ ),
+ ("_global_dim_stride_b16", "override::global_dim_stride", ("2d", "3d",
"4d", "5d"), "b16"),
+)
+
+
+def _fabric_tensor_handle_operands(bracket, handle, override, size_type):
+ """`[tensorMap, leId, off{, counterOff}{, attributeOverrideInfo},
tensorCoords]`.
+
+ The TMA override operands with the fabric handle (logical-endpoint id, byte
+ offset, and the counter offset of counted completion) in place of TMA's
+ `global_address`.
+ """
+ if override is None:
+ attributes = ()
+ coords = OperandSlot("coords", dtype="s32", lanes=_tma_coords_lanes,
bracket=bracket)
+ elif override == "override::global_dim":
+ attributes = (
+ OperandSlot("tensor_size", dtype=size_type, lanes=1, vector=True,
bracket=bracket),
+ )
+ coords = OperandSlot("coords", dtype="s32", lanes=1, vector=True,
bracket=bracket)
+ else:
+ attributes = (
+ OperandSlot("tensor_size", dtype=size_type, lanes=_tma_dim_lanes,
bracket=bracket),
+ OperandSlot(
+ "lower_stride", dtype="b32", lanes=_tma_lower_stride_lanes,
bracket=bracket
+ ),
+ OperandSlot("upper_stride", dtype="b16", bracket=bracket),
+ )
+ coords = OperandSlot("coords", dtype="s32", lanes=_tma_coords_lanes,
bracket=bracket)
+ return (
+ OperandSlot("tmap", kind="addr", space="global", bracket=bracket),
+ *(OperandSlot(name, dtype=dtype, bracket=bracket) for name, dtype in
handle),
+ *attributes,
+ coords,
+ )
+
+
+_FABRIC_ATOM_TYPES = {
+ "and": ("b32", "b64"),
+ "xor": ("b32", "b64"),
+ "or": ("b32", "b64"),
+ "exch": ("b32", "b64", "b128"),
+ "cas": ("b32", "b64", "b128"),
+ "min": ("u32", "u64", "f16x2", "bf16x2"),
+ "max": ("u32", "u64", "f16x2", "bf16x2"),
+ "add": ("u32", "u64", "f16x2", "bf16x2", "f32", "f64"),
+}
+
+
+def _check_fabric_atom(m):
+ """Exact fabric.try_atom operation/type grid (PTX ISA 9.4, 9.7.11.5.10)."""
+ op, ty = m["redop"], m["type"]
+ if ty not in _FABRIC_ATOM_TYPES[op]:
+ allowed = " / ".join("." + item for item in _FABRIC_ATOM_TYPES[op])
+ return f".{op} fabric atomic takes {allowed}"
+ return None
+
+
_TCGEN05_COLLECTOR_A = (
"collector::a::fill",
"collector::a::use",
@@ -3271,6 +3366,13 @@ def _check_set_packed(m):
return None
+def _check_sin_cos_2pi(m):
+ """`.2pi` requires `.ftz` (PTX ISA 9.7.3.18/19, PTX ISA 9.5)."""
+ if m["unit"] and not m["ftz"]:
+ return ".ftz is required when .2pi is specified"
+ return None
+
+
_PTX_94_ENTRIES = [
# PTX ISA 9.4, 9.7.5.1/2 -- SM107 mixed packed add/sub. Keep the
# three type positions independent: the syntax defines one up-conversion
@@ -5724,17 +5826,311 @@ _PTX_94_ENTRIES = [
OperandSlot("p", kind="addr", allow_imm_offset=True),
),
),
+ # fabric.try_{get,put,red}.async.tensor, per PTX ISA 9.4 9.7.11.5.7-9:
+ # the TMA-shaped fabric operations, which address a tensor through a
+ # tensor map plus a fabric handle `[tensorMap, leId, off{, counterOff}
+ # {, attributeOverrideInfo}, tensorCoords]`. `.override::global_address`
+ # is mandatory (the handle replaces the tensor map's global address);
+ # `.override_attribute` follows TMA's shapes (`_FABRIC_TENSOR_OVERRIDES`).
+ # try_get ends in `.b128` like try_put, while try_red ends in its `.redOp`
+ # alone: "the type of each data element ... is specified by the tensorMap".
+ # "Requires sm_107f or higher in the same family." MEASURED on CUDA 13.5:
+ # every shape assembles at sm_107f and sm_107a and is refused at sm_100a,
+ # sm_103a and sm_120f; `.override::global_dim` with `.2d` is refused
+ # ("Illegal modifier"), so is a type on try_red and a form without
+ # `.override::global_address`. ptxas also takes `.L2::cache_hint` after
+ # `.relaxed.sys`, as the ISA's own Examples write it; the entries spell the
+ # syntax line's order.
+ *[
+ InstructionEntry(
+ name=f"fabric_try_get_tensor{suffix}",
+ mnemonic="fabric",
+ slots=(
+ ModifierSlot("action", ("try_get",)),
+ ModifierSlot("api", ("async",)),
+ ModifierSlot("unit", ("tensor",)),
+ ModifierSlot("dim", dims),
+ ModifierSlot("dst", ("shared::cta",)),
+ ModifierSlot("load_mode", ("tile",), optional=True),
+ ModifierSlot("completion", ("mbarrier::complete_tx::bytes",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
+ ModifierSlot("cache", ("L2::cache_hint",), optional=True),
+ ModifierSlot("override_address",
("override::global_address",)),
+ *((ModifierSlot("override_attribute", (override,)),) if
override else ()),
+ ModifierSlot("sem", ("relaxed",)),
+ ModifierSlot("scope", ("sys",)),
+ ModifierSlot("type", ("b128",)),
+ ),
+ cert_arch="sm_107f",
+ operands=(
+ OperandSlot("dst_mem", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ *_fabric_tensor_handle_operands(
+ "src", (("src_le_id", "u32"), ("src_data_off", "u64")),
override, size_type
+ ),
+ OperandSlot("mbar", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("cache_policy", dtype="u64",
lanes=_tma_cache_lanes, vector=False),
+ ),
+ )
+ for suffix, override, dims, size_type in _FABRIC_TENSOR_OVERRIDES
+ ],
+ *[
+ InstructionEntry(
+ name=f"fabric_try_{kind}_tensor{'_counted' if counted else
''}{suffix}",
+ mnemonic="fabric",
+ slots=(
+ ModifierSlot("action", (f"try_{kind}",)),
+ ModifierSlot("api", ("async",)),
+ ModifierSlot("unit", ("tensor",)),
+ ModifierSlot("multimem", ("multimem",), optional=True),
+ ModifierSlot("dim", dims),
+ ModifierSlot("src", ("shared::cta",)),
+ ModifierSlot("load_mode", ("tile",), optional=True),
+ ModifierSlot("completion", ("mbarrier::complete_tx::16B",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
+ *((ModifierSlot("counted", ("counted::bytes",)),) if counted
else ()),
+ ModifierSlot("cache", ("L2::cache_hint",), optional=True),
+ ModifierSlot("override_address",
("override::global_address",)),
+ *((ModifierSlot("override_attribute", (override,)),) if
override else ()),
+ ModifierSlot("sem", ("relaxed",)),
+ ModifierSlot("scope", ("sys",)),
+ (
+ ModifierSlot("type", ("b128",))
+ if kind == "put"
+ else ModifierSlot("redop", ("and", "xor", "or", "add",
"min", "max"))
+ ),
+ ),
+ cert_arch="sm_107f",
+ operands=(
+ *_fabric_tensor_handle_operands(
+ "dst",
+ (
+ ("dst_le_id", "u32"),
+ ("dst_data_off", "u64"),
+ *((("dst_counter_off", "u64"),) if counted else ()),
+ ),
+ override,
+ size_type,
+ ),
+ OperandSlot("src_mem", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("size", dtype="u32"),
+ OperandSlot("mbar", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("cache_policy", dtype="u64",
lanes=_tma_cache_lanes, vector=False),
+ ),
+ )
+ for kind in ("put", "red")
+ for counted in (False, True)
+ for suffix, override, dims, size_type in _FABRIC_TENSOR_OVERRIDES
+ ],
+ # fabric.try_atom, per PTX ISA 9.4 9.7.11.5.10: an atomic on a fabric
+ # handle that leaves the previous value in shared memory. Four syntax
+ # lines that differ only in their operation/type pair, so one entry whose
+ # `_check_fabric_atom` grid is the ISA's `.redOp*`/`.type*` sets. "Requires
+ # sm_100 or higher." MEASURED on CUDA 13.5: every listed pair assembles at
+ # sm_100 (and sm_100a, sm_103a, sm_107f), sm_90a refuses the instruction,
+ # and pairs outside the grid are refused ("Incorrect type '.s32' for
+ # operation '.add'") -- except `.min/.max.s32`, which ptxas accepts though
+ # the ISA does not list it; the table follows the ISA. ptxas also takes
+ # `.L2::cache_hint` before the completion mechanism, as the ISA's third
+ # Example writes it; the entry spells the syntax line's order.
+ InstructionEntry(
+ name="fabric_try_atom",
+ mnemonic="fabric",
+ slots=(
+ ModifierSlot("action", ("try_atom",)),
+ ModifierSlot("api", ("async",)),
+ ModifierSlot("src", ("shared::cta",)),
+ ModifierSlot("completion", ("mbarrier::complete_tx::16B",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
+ ModifierSlot("cache", ("L2::cache_hint",), optional=True),
+ ModifierSlot("sem", ("relaxed",)),
+ ModifierSlot("scope", ("sys",)),
+ ModifierSlot("redop", tuple(_FABRIC_ATOM_TYPES)),
+ ModifierSlot(
+ "type", ("b32", "b64", "b128", "u32", "u64", "f16x2",
"bf16x2", "f32", "f64")
+ ),
+ ),
+ check=_check_fabric_atom,
+ cert_arch="sm_100",
+ operands=(
+ OperandSlot("dst_le_id", dtype="u32", bracket="dst"),
+ OperandSlot("dst_data_off", dtype="u64", bracket="dst"),
+ OperandSlot("smem_dst", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("smem_src", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("mbar", kind="addr", allow_imm_offset=True,
space="shared::cta"),
+ OperandSlot("cache_policy", dtype="u64", lanes=_tma_cache_lanes,
vector=False),
+ ),
+ ),
+]
+
+# PTX ISA 9.5 additions, grouped by ISA release like `_PTX_94_ENTRIES`. Every
+# MEASURED note in this group records CUDA 13.5 ptxas behaviour, and ptxas
+# refuses these spellings under `.version 9.4` ("Feature ... requires PTX ISA
+# .version 9.5 or later").
+_PTX_95_ENTRIES = [
+ # fabric.submit.commit_group / fabric.wait.sync_restrict::read N, per PTX
+ # ISA 9.7.11.5.5 and 9.7.11.5.6 (both "Requires sm_100 or higher").
+ #
+ # A fabric submit group tracks the submit of every prior fabric operation
+ # of the issuing thread; `fabric.wait.sync_restrict::read N` waits until at
+ # most N of the most recent groups are pending, after which the
+ # .shared::cta data those groups read may be overwritten -- the same
+ # commit/wait shape as cp.async.bulk. They are entries of their own rather
+ # than extra choices on the PTX 9.3 `fabric_submit`/`fabric_wait`: they are
+ # separate syntax lines (the wait gains an operand), and the 9.3 entries
+ # keep their own sm_103a certification manifest.
+ #
+ # N is an integer constant with no ISA-declared domain, kept OPEN like
+ # cp.async.bulk.wait_group's. MEASURED: N = 0 and N = 255 assemble at
+ # sm_100, sm_100a, sm_103a and sm_107f; sm_90 and sm_90a refuse the
+ # instruction.
+ InstructionEntry( # fabric.submit.commit_group;
+ name="fabric_submit_commit_group",
+ mnemonic="fabric",
+ slots=(
+ ModifierSlot("action", ("submit",)),
+ ModifierSlot("group", ("commit_group",)),
+ ),
+ cert_arch="sm_100",
+ orders_memory=True,
+ operands=(),
+ ),
+ InstructionEntry( # fabric.wait.sync_restrict::read N;
+ name="fabric_wait_read",
+ mnemonic="fabric",
+ slots=(
+ ModifierSlot("action", ("wait",)),
+ ModifierSlot("waitop", ("sync_restrict::read",)),
+ ),
+ cert_arch="sm_100",
+ orders_memory=True,
+ operands=(OperandSlot("group", kind="imm"),),
+ ),
+ # cp.async.bulk.wait_group.sync_restrict::proxy::async N, per PTX ISA
+ # 9.7.10.28.6.2: the qualifier "only orders async-proxy global and shared
+ # memory accesses before subsequent async-proxy accesses; not before
+ # subsequent generic-proxy accesses". It is its own syntax line, without
+ # `{.read}`: MEASURED, ptxas answers "Invalid combination of
+ # '.sync_restrict::proxy::async' and '.read'" at every architecture.
+ #
+ # The ISA requires sm_100 or higher. MEASURED: ptxas also assembles it at
+ # sm_90 and sm_90a; the table follows the ISA where ptxas is the more
+ # permissive of the two, so this entry certifies at sm_100.
+ InstructionEntry( # cp.async.bulk.wait_group.sync_restrict::proxy::async
N;
+ name="cp_async_bulk_wait_group_proxy_async",
+ mnemonic="cp",
+ slots=(
+ ModifierSlot("api", ("async",)),
+ ModifierSlot("kind", ("bulk",)),
+ ModifierSlot("action", ("wait_group",)),
+ ModifierSlot("sync_restrict", ("sync_restrict::proxy::async",)),
+ ),
+ cert_arch="sm_100",
+ orders_memory=True,
+ operands=(OperandSlot("group", kind="imm"),),
+ ),
+ # fence.alias.sync_restrict::read.release.sys, per PTX ISA 9.7.15.4: the
+ # generic-proxy to generic-proxy alias fence ("requires sm_90 or higher").
+ # A fixed token sequence. MEASURED: ptxas refuses any other scope
+ # ("Modifier '.sync_restrict::read' cannot be combined with modifier
+ # '.gpu'").
+ InstructionEntry( # fence.alias.sync_restrict::read.release.sys;
+ name="fence_alias_sync_restrict_read",
+ mnemonic="fence",
+ slots=(
+ ModifierSlot("proxykind", ("alias",)),
+ ModifierSlot("sync_restrict", ("sync_restrict::read",)),
+ ModifierSlot("sem", ("release",)),
+ ModifierSlot("scope", ("sys",)),
+ ),
+ cert_arch="sm_90",
+ orders_memory=True,
+ operands=(),
+ ),
+ # tensormap.replace `{.alloc_class}` and tensormap.query.alloc_class_large,
+ # per PTX ISA 9.7.10.29/30: the Tensor Allocation Size of a tensor-map
+ # object, small or large. Two new tensormap.replace syntax lines --
+ # tensormap.replace.mode.field4{.alloc_class}{.ss}.b1024.type [addr],
new_val;
+ # tensormap.replace.mode.alloc_class{.ss}.b1024 [addr];
+ # -- where the field-less line takes only .alloc_small/.alloc_large and
+ # `.alloc_same` ("unchanged") exists only beside `.global_address`. The
+ # global_address line without `.alloc_class` stays the pre-9.5
+ # `tensormap_replace_address` entry; PTX ISA 9.5 clarifies that it now sets
+ # the allocation size to Small.
+ #
+ # "Qualifier .alloc_class is supported on sm_90a, and on following
+ # family-specific architectures: sm_100f / sm_110f / sm_120f or higher in
+ # the same family", and the query has the same list. MEASURED: all three
+ # assemble at sm_90a, sm_100a, sm_100f, sm_103a, sm_107a, sm_107f and
+ # sm_120f, and are refused at sm_90 and sm_100; `.alloc_same` without a
+ # field answers "Modifier '.alloc_same' requires modifier
+ # '.global_address'", and the other fields refuse `.alloc_*`.
+ #
+ # NOT REGISTERED: the query's `.const` and `.param{::entry}` state spaces.
+ # Neither address can cross the helper-function ABI (the reason `ld` leaves
+ # out `.param`); a `__grid_constant__` kernel-parameter tensor map is
+ # reached through generic addressing, the no-`.ss` spelling registered
here.
+ # Nor the query's `.shared::cta`: MEASURED on CUDA 13.5 (sm_90a, sm_100a,
+ # sm_107a), ptxas refuses it with a 32-bit address register -- "Arguments
+ # mismatch for instruction 'mov'" -- while accepting a 64-bit register or a
+ # variable name, and the shared-address ABI here is the 32-bit window
+ # offset (the field-less and global_address replaces take it fine). So a
+ # tensor map in shared memory has no query helper yet; the replaces do.
+ InstructionEntry( #
tensormap.replace.tile.global_address.alloc_class{.ss}.b1024.b64
+ name="tensormap_replace_address_alloc_class",
+ mnemonic="tensormap.replace",
+ slots=(
+ ModifierSlot("mode", ("tile",)),
+ ModifierSlot("field", ("global_address",)),
+ ModifierSlot("alloc_class", ("alloc_small", "alloc_large",
"alloc_same")),
+ ModifierSlot("space", ("global", "shared::cta"), optional=True),
+ ModifierSlot("width", ("b1024",)),
+ ModifierSlot("type", ("b64",)),
+ ),
+ cert_arch="sm_90a",
+ operands=(
+ OperandSlot("addr", kind="addr", allow_imm_offset=True),
+ OperandSlot("new_val", dtype="b64"),
+ ),
+ ),
+ InstructionEntry( # tensormap.replace.tile.alloc_class{.ss}.b1024 [addr];
+ name="tensormap_replace_alloc_class",
+ mnemonic="tensormap.replace",
+ slots=(
+ ModifierSlot("mode", ("tile",)),
+ ModifierSlot("alloc_class", ("alloc_small", "alloc_large")),
+ ModifierSlot("space", ("global", "shared::cta"), optional=True),
+ ModifierSlot("width", ("b1024",)),
+ ),
+ cert_arch="sm_90a",
+ operands=(OperandSlot("addr", kind="addr", allow_imm_offset=True),),
+ ),
+ InstructionEntry( # tensormap.query.alloc_class_large{.ss}.b1024 p,
[addr];
+ name="tensormap_query",
+ mnemonic="tensormap.query",
+ slots=(
+ ModifierSlot("query", ("alloc_class_large",)),
+ ModifierSlot("space", ("global",), optional=True),
+ ModifierSlot("width", ("b1024",)),
+ ),
+ cert_arch="sm_90a",
+ operands=(
+ OperandSlot("p", rw="w", dtype="pred"),
+ OperandSlot("addr", kind="addr", allow_imm_offset=True),
+ ),
+ ),
]
_ENTRIES = [
*_PTX_94_ENTRIES,
+ *_PTX_95_ENTRIES,
# ------------------------------------------------------------------
# PTX ISA 9.7.1 — Integer Arithmetic Instructions
# ------------------------------------------------------------------
- # The existing public families cover this section with one exception,
- # stated at the entry it belongs to: the packed `.u8x4`/`.s8x4` types
- # (and the saturating forms that arrived with them), which the ISA gives
- # only to "sm_120f or higher in the same family". The distinct `clmad`
+ # The existing public families cover this section. The packed
+ # `.u8x4`/`.s8x4` types (and the saturating forms that arrived with them),
+ # which the ISA gives only to "sm_107f or higher in the same family,
sm_120f
+ # or higher in the same family", are sm_107f sibling entries beside the
+ # families they extend, so those families keep their floors. The distinct
`clmad`
# family (9.7.1.5, introduced in PTX ISA 9.3) is registered below.
#
# Several mnemonics have a floating-point line further down the table. The
@@ -5747,9 +6143,9 @@ _ENTRIES = [
# (single/double), 9.7.4.7, 9.7.4.8 (half).
# Each mnemonic gets two entries because the ISA gives it two operand
# shapes: the usual `d, a, b` and the three-source `d, a, b, c` line.
- # NOT REGISTERED: only the `.u8x4` and `{.relu}.s8x4` tokens of the integer
- # lines (reason at the `_MINMAX_INT_PLAIN` note above: sm_120f-only). Every
- # other syntax line and type token of both families is here.
+ # The `.u8x4` and `{.relu}.s8x4` tokens of the integer lines are the
sm_107f
+ # `{min,max}_int8x4` entries that follow; every syntax line and type token
+ # of both families is registered.
*[
InstructionEntry(
name=name,
@@ -5773,6 +6169,29 @@ _ENTRIES = [
)
for name in ("max", "min")
],
+ # `min/max.u8x4` and `min/max{.relu}.s8x4` (9.7.1.13/9.7.1.14), given only
+ # to "sm_107f or higher in the same family, sm_120f or higher in the same
+ # family": a sibling entry certified at sm_107f. MEASURED on CUDA 13.5:
+ # assembles at sm_107a, sm_107f and sm_120f, refused at sm_90, sm_100a and
+ # sm_103a; `.relu.u8x4` is refused everywhere.
+ *[
+ InstructionEntry(
+ name=f"{name}_int8x4",
+ mnemonic=name,
+ slots=(
+ ModifierSlot("relu", ("relu",), optional=True),
+ ModifierSlot("type", ("u8x4", "s8x4")),
+ ),
+ check=_check_minmax_int8x4,
+ cert_arch="sm_107f",
+ operands=(
+ OperandSlot("d", rw="w"),
+ OperandSlot("a"),
+ OperandSlot("b"),
+ ),
+ )
+ for name in ("max", "min")
+ ],
*[
InstructionEntry(
name=f"{name}3",
@@ -5826,9 +6245,9 @@ _ENTRIES = [
# sub{.sat}.type2 d, a, b; .type2 = {.s32, .u8x4, .s8x4}
# The two lines differ only in whether `.sat` is offered, so one entry with
# an optional slot spells both and `_check_int_addsub` draws the boundary.
- # NOT REGISTERED: the `.u8x4`/`.s8x4` types on either mnemonic, and `.sat`
- # on .u32/.u16x2/.s16x2 -- all sm_120f-only (PTX ISA 9.2), the same
- # exclusion the min/max entry above makes for the same reason.
+ # The `.u8x4`/`.s8x4` types on either mnemonic, and `.sat` on
+ # .u32/.u16x2/.s16x2, are the sm_107f `add_int_sm107f`/`sub_int_sm107f`
+ # entries after these two.
InstructionEntry(
name="add_int",
mnemonic="add",
@@ -5860,6 +6279,43 @@ _ENTRIES = [
OperandSlot("b"),
),
),
+ # The integer add/sub forms the ISA gives only to "sm_107f or higher in the
+ # same family, sm_120f or higher in the same family" (9.7.1.1/9.7.1.2
Target
+ # ISA Notes, PTX ISA 9.2): the packed `.u8x4`/`.s8x4` types with or without
+ # `.sat`, and `.sat` on .u32/.u16x2/.s16x2. Sibling entries certified at
+ # sm_107f, so `add_int`/`sub_int` keep their floors. MEASURED on CUDA 13.5:
+ # every one of these assembles at sm_107a, sm_107f and sm_120f and is
refused
+ # at sm_90, sm_100a and sm_103a; `sub.sat.u32` is refused everywhere, as
+ # the syntax (`sub{.sat}.type2`, `.type2 = {.s32, .u8x4, .s8x4}`) says.
+ InstructionEntry(
+ name="add_int_sm107f",
+ mnemonic="add",
+ slots=(
+ ModifierSlot("sat", ("sat",), optional=True),
+ ModifierSlot("type", ("u32", "u16x2", "u8x4", "s16x2", "s8x4")),
+ ),
+ check=_check_int_add_sm107f,
+ cert_arch="sm_107f",
+ operands=(
+ OperandSlot("d", rw="w"),
+ OperandSlot("a"),
+ OperandSlot("b"),
+ ),
+ ),
+ InstructionEntry(
+ name="sub_int_sm107f",
+ mnemonic="sub",
+ slots=(
+ ModifierSlot("sat", ("sat",), optional=True),
+ ModifierSlot("type", ("u8x4", "s8x4")),
+ ),
+ cert_arch="sm_107f",
+ operands=(
+ OperandSlot("d", rw="w"),
+ OperandSlot("a"),
+ OperandSlot("b"),
+ ),
+ ),
# mul / mad per PTX ISA 9.7.1.3, 9.7.1.4:
# mul.mode.type d, a, b; mad.mode.type d, a, b, c;
# .mode = {.hi, .lo, .wide} mad.hi.sat.s32 d, a, b, c;
@@ -6002,7 +6458,7 @@ _ENTRIES = [
# abs / neg per PTX ISA 9.7.1.11, 9.7.1.12 -- signed integers only. Their
# floating-point lines (9.7.3.9, 9.7.3.10) are the `abs_f` and `neg`
# entries below; which of each pair carries a suffix is just which arrived
- # second. NOT REGISTERED: `neg.s8x4`, sm_120f-only as above.
+ # second. `neg.s8x4` is the sm_107f `neg_int_sm107f` entry after them.
InstructionEntry(
name="abs",
slots=(ModifierSlot("type", ("s16", "s32", "s64")),),
@@ -6020,6 +6476,20 @@ _ENTRIES = [
OperandSlot("a"),
),
),
+ # `neg.s8x4` (9.7.1.12), "sm_107f or higher in the same family, sm_120f or
+ # higher in the same family" -- the sm_107f sibling of `neg_int`. MEASURED
+ # on CUDA 13.5 as for the packed add/sub above; `neg.u8x4` and `abs.s8x4`
+ # are not in the syntax and ptxas refuses them.
+ InstructionEntry(
+ name="neg_int_sm107f",
+ mnemonic="neg",
+ slots=(ModifierSlot("type", ("s8x4",)),),
+ cert_arch="sm_107f",
+ operands=(
+ OperandSlot("d", rw="w"),
+ OperandSlot("a"),
+ ),
+ ),
# popc / clz per PTX ISA 9.7.1.15, 9.7.1.16: `popc.type d, a;`. Both count
# bits of a `.b32`/`.b64` source into a `.u32` destination whatever the
# source width -- "destination d has type .u32" -- so d is typed outright
@@ -6340,20 +6810,36 @@ _ENTRIES = [
# half-precision forms (9.7.4.10) are the `ex2_half` entry below: they
# cannot share this optional `.ftz` slot, because theirs is mandatory on
# the bf16 line and unspelled on the f16 one.
+ #
+ # PTX ISA 9.5 adds `{.2pi}` to sin and cos only --
`op.approx{.2pi}{.ftz}.f32`,
+ # reading the input in cycles (1.0 is one full turn) -- and "the .ftz
+ # qualifier is required when .2pi is specified", which `_check_sin_cos_2pi`
+ # states. It is a qualifier on the same syntax line, so it is a slot here
+ # rather than an entry in `_PTX_95_ENTRIES`, and the helper signature of
+ # `T.ptx.sin`/`T.ptx.cos` stays the one shape it was. `2pi` is not a
Python
+ # identifier, so programs spell it through the string form
+ # (`T.ptx["sin.approx.2pi.ftz.f32"]`), as they do the TMA `1d`..`5d` dims.
+ # "Supported on all target architectures." MEASURED on CUDA 13.5:
assembles
+ # at sm_75, sm_90, sm_100 and sm_107a; without `.ftz` ptxas answers ".ftz
+ # modifier required for instruction 'sin' with modifier '.2pi'"; lg2 and
ex2
+ # refuse `.2pi` ("Illegal modifier"); `.version 9.4` refuses it.
*[
InstructionEntry(
name=name,
slots=(
ModifierSlot("mode", ("approx",)),
+ *((ModifierSlot("unit", ("2pi",), optional=True),) if trig
else ()),
ModifierSlot("ftz", ("ftz",), optional=True),
ModifierSlot("type", ("f32",)),
),
+ check=_check_sin_cos_2pi if trig else None,
operands=(
OperandSlot("d", rw="w"),
OperandSlot("value"),
),
)
for name in ("ex2", "sin", "cos", "lg2")
+ for trig in (name in ("sin", "cos"),)
],
# tanh per PTX ISA 9.7.3.22: `tanh.approx.f32 d, a;` -- alone among the
# approximations in spelling no `.ftz`, so it gets no such slot. Requires
@@ -7479,7 +7965,8 @@ _ENTRIES = [
# enforces by name.
# NOT REGISTERED: `.swizzle_atomicity` (sm_100a and later only, while the
# rest of this instruction is sm_90a), and the `.elemtype` values 13-15 and
- # `.swizzle_mode` value 4, which need newer architectures still.
+ # `.swizzle_mode` value 4, which need newer architectures still. PTX ISA
+ # 9.5's `.alloc_class` lines and tensormap.query are in `_PTX_95_ENTRIES`.
*[
InstructionEntry(
name=f"tensormap_replace_{group}",
@@ -9029,6 +9516,11 @@ _ENTRIES = [
# PTX ISA 9.7.11 — Fabric Instructions (introduced in PTX ISA 9.3)
# ------------------------------------------------------------------
# A fabric handle is the composite [logical-endpoint id, byte offset].
+ # The completion mechanism `.mbarrier::complete_tx::{bytes,16B}
+ # .mbarrier::report::fabric{.counted::bytes}` is two or three dotted
+ # qualifiers, so it is that many slots: a single slot holding the dotted
+ # text renders the same PTX but cannot be spelled by `T.ptx[...]` or an
+ # attribute chain, which split on the dots.
# Counted completion adds a third counter-offset member and is therefore a
# distinct operand shape. Likewise, cp_mask has a trailing byte-mask
# operand and is incompatible with counted::bytes (ISA 9.7.11.5.2).
@@ -9039,7 +9531,8 @@ _ENTRIES = [
ModifierSlot("action", ("try_get",)),
ModifierSlot("api", ("async",)),
ModifierSlot("dst", ("shared::cta",)),
- ModifierSlot("completion",
("mbarrier::complete_tx::bytes.mbarrier::report::fabric",)),
+ ModifierSlot("completion", ("mbarrier::complete_tx::bytes",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
ModifierSlot("sem", ("relaxed",)),
ModifierSlot("scope", ("sys",)),
ModifierSlot("type", ("b128",)),
@@ -9062,13 +9555,9 @@ _ENTRIES = [
ModifierSlot("api", ("async",)),
ModifierSlot("multimem", ("multimem",), optional=True),
ModifierSlot("src", ("shared::cta",)),
- ModifierSlot(
- "completion",
- (
- "mbarrier::complete_tx::16B.mbarrier::report::fabric"
- + (".counted::bytes" if counted else ""),
- ),
- ),
+ ModifierSlot("completion", ("mbarrier::complete_tx::16B",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
+ *((ModifierSlot("counted", ("counted::bytes",)),) if counted
else ()),
*((ModifierSlot("cp_mask", ("cp_mask",)),) if cp_mask else ()),
ModifierSlot("sem", ("relaxed",)),
ModifierSlot("scope", ("sys",)),
@@ -9100,13 +9589,9 @@ _ENTRIES = [
ModifierSlot("api", ("async",)),
ModifierSlot("multimem", ("multimem",), optional=True),
ModifierSlot("src", ("shared::cta",)),
- ModifierSlot(
- "completion",
- (
- "mbarrier::complete_tx::16B.mbarrier::report::fabric"
- + (".counted::bytes" if counted else ""),
- ),
- ),
+ ModifierSlot("completion", ("mbarrier::complete_tx::16B",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
+ *((ModifierSlot("counted", ("counted::bytes",)),) if counted
else ()),
ModifierSlot("sem", ("relaxed",)),
ModifierSlot("scope", ("sys",)),
ModifierSlot("redop", tuple(_FABRIC_RED_TYPES)),
@@ -9139,7 +9624,8 @@ _ENTRIES = [
ModifierSlot("api", ("async",)),
ModifierSlot("multimem", ("multimem",)),
ModifierSlot("dst", ("shared::cta",)),
- ModifierSlot("completion",
("mbarrier::complete_tx::bytes.mbarrier::report::fabric",)),
+ ModifierSlot("completion", ("mbarrier::complete_tx::bytes",)),
+ ModifierSlot("report", ("mbarrier::report::fabric",)),
ModifierSlot("sem", ("relaxed",)),
ModifierSlot("scope", ("sys",)),
ModifierSlot("redop", tuple(_FABRIC_PULLRED_TYPES)),
@@ -9168,6 +9654,10 @@ _ENTRIES = [
orders_memory=True,
operands=(),
),
+ # `fabric.wait.sync_restrict::reads` is deprecated by PTX ISA 9.5 in
+ # favour of `fabric.wait.sync_restrict::read N` with
+ # `fabric.submit.commit_group` (both in `_PTX_95_ENTRIES`), and "has no
+ # effect" (9.7.11.5.6). Kept so existing programs keep their spelling.
InstructionEntry(
name="fabric_wait",
mnemonic="fabric",
@@ -9486,9 +9976,10 @@ _ENTRIES = [
# alone is not enough. Each syntax line whose token sequence differs is its
# own entry, all sharing the `fence` mnemonic, so the surface stays
# `T.ptx.fence...` and the chain narrows on the tokens themselves.
- # NOT REGISTERED: the `.sync_restrict` lines (fixed token sequences with no
- # users yet), and the deprecated `membar` spellings, which the ISA itself
- # marks as the old style for `fence`.
+ # NOT REGISTERED: the pre-9.5 `.sync_restrict` lines (fixed token
+ # sequences with no users yet), and the deprecated `membar` spellings,
+ # which the ISA itself marks as the old style for `fence`. PTX ISA 9.5's
+ # `fence.alias.sync_restrict::read.release.sys` is in `_PTX_95_ENTRIES`.
InstructionEntry( # fence{.sem}.scope;
name="fence",
slots=(
@@ -11211,10 +11702,26 @@ _ENTRIES = [
# Every one of these carries `orders_memory=True`: the legacy helpers all
# had `::: "memory"`, and the waits/fences name no address at all.
#
- # NOT REGISTERED:
- # - `tcgen05.shift.cta_group.down [taddr]`: no call site. (The tmem address
- # space its bracketed operand needs now exists --
`tcgen05.ld`/`.st`/`.cp`
- # all use it -- so this one is registrable the day something calls it.)
+ #
+ # tcgen05.shift per PTX ISA 9.7.18.9.3: asynchronously shift the rows of a
+ # tmem matrix down by one, completed through tcgen05.commit like
+ # tcgen05.mma/.cp. Supported on sm_100a, sm_103a, sm_107a and sm_110a only
+ # -- no family-specific (`f`) target -- so it certifies at sm_100a.
+ # MEASURED on CUDA 13.5: assembles at sm_100a, sm_103a, sm_107a and sm_110a
+ # and is refused at sm_100f, sm_107f and sm_120a. ptxas also takes the
+ # `.down.cta_group::N` order of the ISA's own Examples; the entry spells
the
+ # syntax line's `.cta_group.down`.
+ InstructionEntry( # tcgen05.shift.cta_group.down [taddr];
+ name="tcgen05_shift",
+ mnemonic="tcgen05",
+ slots=(
+ ModifierSlot("action", ("shift",)),
+ ModifierSlot("cta_group", ("cta_group::1", "cta_group::2")),
+ ModifierSlot("direction", ("down",)),
+ ),
+ cert_arch="sm_100a",
+ operands=(OperandSlot("taddr", kind="addr", space="tmem"),),
+ ),
InstructionEntry( #
tcgen05.alloc.cta_group.sync.aligned{.shared::cta}.b32 [dst], nCols;
name="tcgen05_alloc",
mnemonic="tcgen05",
diff --git a/python/tvm_tirx/script/tirx.pyi b/python/tvm_tirx/script/tirx.pyi
index 8c0d91434..1106ae0fe 100644
--- a/python/tvm_tirx/script/tirx.pyi
+++ b/python/tvm_tirx/script/tirx.pyi
@@ -52,7 +52,7 @@ class _Chain_activemask:
) -> None: ...
class _Chain_add:
- """`add` — 6 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`add` — 7 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (d, a, c); (d, a, b)
"""
@@ -72,11 +72,13 @@ class _Chain_add:
s16x2: _Chain_add
s32: _Chain_add
s64: _Chain_add
+ s8x4: _Chain_add
sat: _Chain_add
u16: _Chain_add
u16x2: _Chain_add
u32: _Chain_add
u64: _Chain_add
+ u8x4: _Chain_add
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_and:
@@ -373,7 +375,9 @@ class _Chain_copysign:
) -> None: ...
class _Chain_cos:
- """`cos` — mode∈{approx}; ftz∈{ftz} (opt); type∈{f32}"""
+ """`cos` — mode∈{approx}; unit∈{2pi} (opt); ftz∈{ftz} (opt); type∈{f32} —
`.2pi` requires
+ `.ftz` (PTX ISA 9.7.3.18/19, PTX ISA 9.5).
+ """
approx: _Chain_cos
f32: _Chain_cos
@@ -388,7 +392,7 @@ class _Chain_cos:
) -> None: ...
class _Chain_cp:
- """`cp` — 77 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`cp` — 78 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (*__operands); (src_mem, size); (tmap,
global_address,
tensor_size, coords); (group); (); (dst_mem, src_mem, size, mbar); (addr)
"""
@@ -458,6 +462,7 @@ class _Chain_cp:
shared: _Chain_cp
shared__cluster: _Chain_cp
shared__cta: _Chain_cp
+ sync_restrict__proxy__async: _Chain_cp
sys: _Chain_cp
tensor: _Chain_cp
tile: _Chain_cp
@@ -669,12 +674,14 @@ class _Chain_ex2:
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_fabric:
- """`fabric` — 9 entries sharing this mnemonic; PTX puts their difference
in the operand
- list, so the call selects one. Shapes: (dst_mem, src_le_id, src_data_off,
size, mbar);
- (dst_le_id, dst_data_off, src_mem, size, mbar); (dst_le_id, dst_data_off,
src_mem, size,
- mbar, byte_mask); (dst_le_id, dst_data_off, dst_counter_off, src_mem,
size, mbar); ()
+ """`fabric` — 37 entries sharing this mnemonic; PTX puts their difference
in the operand
+ list, so the call selects one. Shapes: (*__operands); (); (group);
(dst_mem, src_le_id,
+ src_data_off, size, mbar); (dst_le_id, dst_data_off, src_mem, size, mbar);
(dst_le_id,
+ dst_data_off, src_mem, size, mbar, byte_mask); (dst_le_id, dst_data_off,
+ dst_counter_off, src_mem, size, mbar)
"""
+ L2__cache_hint: _Chain_fabric
add: _Chain_fabric
and_: _Chain_fabric
async_: _Chain_fabric
@@ -682,23 +689,39 @@ class _Chain_fabric:
b32: _Chain_fabric
b64: _Chain_fabric
bf16: _Chain_fabric
+ bf16x2: _Chain_fabric
+ cas: _Chain_fabric
+ commit_group: _Chain_fabric
+ counted__bytes: _Chain_fabric
cp_mask: _Chain_fabric
+ exch: _Chain_fabric
f16: _Chain_fabric
+ f16x2: _Chain_fabric
f32: _Chain_fabric
f64: _Chain_fabric
max: _Chain_fabric
+ mbarrier__complete_tx__16B: _Chain_fabric
+ mbarrier__complete_tx__bytes: _Chain_fabric
+ mbarrier__report__fabric: _Chain_fabric
min: _Chain_fabric
multimem: _Chain_fabric
op_restrict__fetching: _Chain_fabric
or_: _Chain_fabric
+ override__global_address: _Chain_fabric
+ override__global_dim: _Chain_fabric
+ override__global_dim_stride: _Chain_fabric
relaxed: _Chain_fabric
s32: _Chain_fabric
s64: _Chain_fabric
shared__cta: _Chain_fabric
submit: _Chain_fabric
sync: _Chain_fabric
+ sync_restrict__read: _Chain_fabric
sync_restrict__reads: _Chain_fabric
sys: _Chain_fabric
+ tensor: _Chain_fabric
+ tile: _Chain_fabric
+ try_atom: _Chain_fabric
try_get: _Chain_fabric
try_pullred: _Chain_fabric
try_put: _Chain_fabric
@@ -710,7 +733,7 @@ class _Chain_fabric:
def __call__(self, *args: Any, pred: Any = None) -> None: ...
class _Chain_fence:
- """`fence` — 6 entries sharing this mnemonic; PTX puts their difference in
the operand
+ """`fence` — 7 entries sharing this mnemonic; PTX puts their difference in
the operand
list, so the call selects one. Shapes: (); (addr)
"""
@@ -731,6 +754,7 @@ class _Chain_fence:
sc: _Chain_fence
shared__cluster: _Chain_fence
shared__cta: _Chain_fence
+ sync_restrict__read: _Chain_fence
sys: _Chain_fence
tensormap__generic: _Chain_fence
def __call__(self, *args: Any, pred: Any = None) -> None: ...
@@ -1015,7 +1039,7 @@ class _Chain_match:
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_max:
- """`max` — 2 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`max` — 3 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (d, a, b); (d, a, b, c)
"""
@@ -1033,10 +1057,12 @@ class _Chain_max:
s16x2: _Chain_max
s32: _Chain_max
s64: _Chain_max
+ s8x4: _Chain_max
u16: _Chain_max
u16x2: _Chain_max
u32: _Chain_max
u64: _Chain_max
+ u8x4: _Chain_max
xorsign: _Chain_max
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
@@ -1080,7 +1106,7 @@ class _Chain_mbarrier:
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_min:
- """`min` — 2 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`min` — 3 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (d, a, b); (d, a, b, c)
"""
@@ -1098,10 +1124,12 @@ class _Chain_min:
s16x2: _Chain_min
s32: _Chain_min
s64: _Chain_min
+ s8x4: _Chain_min
u16: _Chain_min
u16x2: _Chain_min
u32: _Chain_min
u64: _Chain_min
+ u8x4: _Chain_min
xorsign: _Chain_min
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
@@ -1417,7 +1445,7 @@ class _Chain_multimem_st_async:
def __call__(self, addr: Any, b: Any, *args: Any, pred: Any = None) ->
None: ...
class _Chain_neg:
- """`neg` — 3 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`neg` — 4 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (d, a)
"""
@@ -1431,6 +1459,7 @@ class _Chain_neg:
s16: _Chain_neg
s32: _Chain_neg
s64: _Chain_neg
+ s8x4: _Chain_neg
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_not:
@@ -1896,7 +1925,9 @@ class _Chain_shr:
) -> None: ...
class _Chain_sin:
- """`sin` — mode∈{approx}; ftz∈{ftz} (opt); type∈{f32}"""
+ """`sin` — mode∈{approx}; unit∈{2pi} (opt); ftz∈{ftz} (opt); type∈{f32} —
`.2pi` requires
+ `.ftz` (PTX ISA 9.7.3.18/19, PTX ISA 9.5).
+ """
approx: _Chain_sin
f32: _Chain_sin
@@ -2124,7 +2155,7 @@ class _Chain_stmatrix:
def __call__(self, *args: Any, pred: Any = None) -> None: ...
class _Chain_sub:
- """`sub` — 6 entries sharing this mnemonic; PTX puts their difference in
the operand list,
+ """`sub` — 7 entries sharing this mnemonic; PTX puts their difference in
the operand list,
so the call selects one. Shapes: (d, a, c); (d, a, b)
"""
@@ -2143,10 +2174,12 @@ class _Chain_sub:
s16: _Chain_sub
s32: _Chain_sub
s64: _Chain_sub
+ s8x4: _Chain_sub
sat: _Chain_sub
u16: _Chain_sub
u32: _Chain_sub
u64: _Chain_sub
+ u8x4: _Chain_sub
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_szext:
@@ -2180,7 +2213,7 @@ class _Chain_tanh:
def __call__(self, *args: Any, pred: Any = None, preserve_dst: bool =
False) -> None: ...
class _Chain_tcgen05:
- """`tcgen05` — 59 entries sharing this mnemonic; PTX puts their difference
in the operand
+ """`tcgen05` — 60 entries sharing this mnemonic; PTX puts their difference
in the operand
list, so the call selects one. Shapes: (dst, ncols); (taddr, ncols);
(mbar, cta_mask);
(mbar); (*__operands); (d_tmem, a_desc, b_desc, idesc, enable_input_d);
(d_tmem, a_desc,
b_desc, idesc, enable_input_d, zero_col_mask); (d_tmem, a_tmem, b_desc,
idesc,
@@ -2194,8 +2227,8 @@ class _Chain_tcgen05:
(d_tmem, a_desc, b_desc, idesc, sfa_tmem, sfb_tmem, enable_input_d);
(d_tmem, a_tmem,
b_desc, idesc, sfa_tmem, sfb_tmem, enable_input_d); (d_tmem, a_desc,
b_desc,
sp_meta_tmem, idesc, sfa_tmem, sfb_tmem, enable_input_d); (d_tmem, a_tmem,
b_desc,
- sp_meta_tmem, idesc, sfa_tmem, sfb_tmem, enable_input_d); (); (taddr,
s_desc); (mbar,
- mask)
+ sp_meta_tmem, idesc, sfa_tmem, sfb_tmem, enable_input_d); (taddr); ();
(taddr, s_desc);
+ (mbar, mask)
"""
NaN: _Chain_tcgen05
@@ -2242,6 +2275,7 @@ class _Chain_tcgen05:
cta_group__2: _Chain_tcgen05
dealloc: _Chain_tcgen05
decompress__lut__b: _Chain_tcgen05
+ down: _Chain_tcgen05
exclusive: _Chain_tcgen05
f32: _Chain_tcgen05
fence__after_thread_sync: _Chain_tcgen05
@@ -2271,6 +2305,7 @@ class _Chain_tcgen05:
scale_vec__4X: _Chain_tcgen05
shared__cluster: _Chain_tcgen05
shared__cta: _Chain_tcgen05
+ shift: _Chain_tcgen05
sp: _Chain_tcgen05
sp__2__colon__4: _Chain_tcgen05
spcompress: _Chain_tcgen05
@@ -2312,11 +2347,30 @@ class _Chain_tensormap_cp_fenceproxy:
tensormap__generic: _Chain_tensormap_cp_fenceproxy
def __call__(self, dst_mem: Any, src_mem: Any, *args: Any, pred: Any =
None) -> None: ...
+class _Chain_tensormap_query:
+ """`tensormap_query` — query∈{alloc_class_large}; space∈{global} (opt);
width∈{b1024}"""
+
+ alloc_class_large: _Chain_tensormap_query
+ b1024: _Chain_tensormap_query
+ global_: _Chain_tensormap_query
+ def __call__(
+ self,
+ p: Any,
+ addr: Any,
+ *args: Any,
+ pred: Any = None,
+ preserve_dst: bool = False,
+ ) -> None: ...
+
class _Chain_tensormap_replace:
- """`tensormap_replace` — 9 entries sharing this mnemonic; PTX puts their
difference in the
- operand list, so the call selects one. Shapes: (addr, new_val); (addr,
ord, new_val)
+ """`tensormap_replace` — 11 entries sharing this mnemonic; PTX puts their
difference in the
+ operand list, so the call selects one. Shapes: (addr, new_val); (addr);
(addr, ord,
+ new_val)
"""
+ alloc_large: _Chain_tensormap_replace
+ alloc_same: _Chain_tensormap_replace
+ alloc_small: _Chain_tensormap_replace
b1024: _Chain_tensormap_replace
b32: _Chain_tensormap_replace
b64: _Chain_tensormap_replace
@@ -2623,6 +2677,7 @@ class _PTX:
tanh: _Chain_tanh
tcgen05: _Chain_tcgen05
tensormap_cp_fenceproxy: _Chain_tensormap_cp_fenceproxy
+ tensormap_query: _Chain_tensormap_query
tensormap_replace: _Chain_tensormap_replace
testp: _Chain_testp
vote_sync: _Chain_vote_sync
diff --git a/tests/python/tirx/codegen/test_cuda.py
b/tests/python/tirx/codegen/test_cuda.py
index c7c9900bc..bbf487576 100644
--- a/tests/python/tirx/codegen/test_cuda.py
+++ b/tests/python/tirx/codegen/test_cuda.py
@@ -1282,5 +1282,69 @@ def test_uint32_loop_var_runs_correctly():
tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+def _locality_domain_kernel():
+ @T.function
+ def kernel(ID: T.Tensor((128, 32), "int32"), SUP: T.Tensor((128, 32),
"int32")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=128, block=32))
+ bx = T.cuda.block_idx("x")
+ tx = T.cuda.thread_idx("x")
+ ID[bx, tx] = T.cuda.mov_sreg(32, "locality_domain_id")
+ SUP[bx, tx] = T.cuda.mov_sreg(32, "locality_domain_supplemental_sm")
+
+ return kernel
+
+
[email protected](
+ not env.has_nvcc_version(13, 5), reason="locality-domain special registers
are PTX 9.5"
+)
+def test_mov_sreg_locality_domain_codegen():
+ """PTX ISA 9.5's %locality_domain_* registers (10.36/10.37) read through
mov_sreg."""
+ src, _ = _get_source(_locality_domain_kernel())
+ assert 'asm volatile("mov.u32 %0, %locality_domain_id;" : "=r"(x));' in src
+ assert 'asm volatile("mov.u32 %0, %locality_domain_supplemental_sm;" :
"=r"(x));' in src
+
+
[email protected]
[email protected](
+ not env.has_nvcc_version(13, 5), reason="locality-domain special registers
are PTX 9.5"
+)
[email protected](
+ not env.has_cuda() or env.cuda_arch() is None or int(env.cuda_arch()[3:6])
< 100,
+ reason="needs an sm_100+ CUDA GPU",
+)
+def test_mov_sreg_locality_domain_gpu():
+ """The domain id is in [0, LOCALITY_DOMAIN_COUNT) and both registers are
CTA-uniform."""
+ target = tvm_tirx.target.Target({"kind": "cuda", "arch": env.cuda_arch()})
+ mod = tvm_tirx.compile(
+ tvm_tirx.IRModule({"main": _locality_domain_kernel()}), target=target,
pipeline="tirx"
+ )
+ domain_count = None
+ try:
+ from cuda.bindings import driver
+
+ attr =
driver.CUdevice_attribute.CU_DEVICE_ATTRIBUTE_LOCALITY_DOMAIN_COUNT
+ driver.cuInit(0)
+ status, count = driver.cuDeviceGetAttribute(attr, 0)
+ if status == driver.CUresult.CUDA_SUCCESS:
+ domain_count = count
+ except (ImportError, AttributeError):
+ pass
+
+ def run_and_check():
+ dev = tvm_tirx.cuda(0)
+ ids = tvm_tirx.runtime.tensor(np.full((128, 32), -1, np.int32),
device=dev)
+ sup = tvm_tirx.runtime.tensor(np.full((128, 32), -1, np.int32),
device=dev)
+ mod(ids, sup)
+ ids, sup = ids.numpy(), sup.numpy()
+ # "For the lifetime of a cluster [resp. CTA], all threads ... read the
same value."
+ assert (ids == ids[:, :1]).all() and (sup == sup[:, :1]).all()
+ assert ids.min() >= 0
+ if domain_count is not None:
+ assert ids.max() < domain_count
+ assert set(np.unique(sup)) <= {0, 1}
+
+ tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+
+
if __name__ == "__main__":
tvm_tirx.testing.main()
diff --git a/tests/python/tirx/codegen/test_ptx_addr.py
b/tests/python/tirx/codegen/test_ptx_addr.py
index 595bf72d8..fb8f29908 100644
--- a/tests/python/tirx/codegen/test_ptx_addr.py
+++ b/tests/python/tirx/codegen/test_ptx_addr.py
@@ -53,10 +53,10 @@ def test_ptx_addr_registration_and_table_capabilities():
assert "tirx.ptx.addr" in CODEGEN_REGISTRY
addresses = [slot for entry in TABLE.values() for slot in entry.operands
if slot.kind == "addr"]
- assert len(addresses) == 417
- assert sum(slot.allow_imm_offset for slot in addresses) == 253
- assert sum(slot.bracket is not None and not slot.allow_imm_offset for slot
in addresses) == 60
- assert sum(slot.space == "tmem" and not slot.allow_imm_offset for slot in
addresses) == 104
+ assert len(addresses) == 499
+ assert sum(slot.allow_imm_offset for slot in addresses) == 309
+ assert sum(slot.bracket is not None and not slot.allow_imm_offset for slot
in addresses) == 85
+ assert sum(slot.space == "tmem" and not slot.allow_imm_offset for slot in
addresses) == 105
def test_ptx_addr_table_validation_rejects_wrong_operand_classes():
diff --git a/tests/python/tirx/codegen/test_ptx_dialect.py
b/tests/python/tirx/codegen/test_ptx_dialect.py
index a25cad006..ca6694fb9 100644
--- a/tests/python/tirx/codegen/test_ptx_dialect.py
+++ b/tests/python/tirx/codegen/test_ptx_dialect.py
@@ -437,7 +437,9 @@ def test_ptx_93_sm103a_exact_renderings():
action="try_put",
api="async",
src="shared::cta",
-
completion=("mbarrier::complete_tx::16B.mbarrier::report::fabric.counted::bytes"),
+ completion="mbarrier::complete_tx::16B",
+ report="mbarrier::report::fabric",
+ counted="counted::bytes",
sem="relaxed",
scope="sys",
type="b128",
@@ -1227,16 +1229,15 @@ def test_ptx_integer_arithmetic_dispatch():
T.device_entry()
T.ptx.mad.lo.sat.s32(out[0], T.int32(2), T.int32(3), T.int32(4))
- # The unavailable add.sat forms added in PTX 9.2 are distinct from the
- # integer lines on which `.sat` has never been legal. Keep both reasons
- # visible so an invalid spelling is not misreported as an architecture
- # gate. In particular, sub.sat.u32 is not the counterpart of add.sat.u32.
- with pytest.raises(ValueError, match="sm_120f-only"):
-
- @T.function
- def add_sat_sm120(out: T.Tensor((1,), "uint32")):
- T.device_entry()
- T.ptx.add.sat.u32(out[0], T.uint32(2), T.uint32(3))
+ # The add.sat forms PTX 9.2 gave to sm_107f/sm_120f resolve to their
sm_107f
+ # sibling entry. They are distinct from the integer lines on which `.sat`
has
+ # never been legal, whose reason stays visible below so an invalid spelling
+ # is not misreported as an architecture gate. In particular, sub.sat.u32 is
+ # not the counterpart of add.sat.u32.
+ @T.function
+ def add_sat_sm107f(out: T.Tensor((1,), "uint32")):
+ T.device_entry()
+ T.ptx.add.sat.u32(out[0], T.uint32(2), T.uint32(3))
with pytest.raises(ValueError, match=r"not on the add\.u64"):
@@ -3059,6 +3060,669 @@ def test_ptx_94_sm107_arch_floors_and_delta():
assert {TABLE[name].cert_arch for name in noftz_siblings} == {"sm_90"}
+_PTX_95_SYNC_FLOORS = {
+ "fabric_submit_commit_group": "sm_100",
+ "fabric_wait_read": "sm_100",
+ "cp_async_bulk_wait_group_proxy_async": "sm_100",
+ "fence_alias_sync_restrict_read": "sm_90",
+}
+
+
+def test_ptx_95_sync_entries_floors_and_delta():
+ """PTX 9.5 commit/wait/fence lines live in the 9.5 delta at their ISA
floor."""
+ from tvm_tirx.backend.cuda.ptx.table import _PTX_95_ENTRIES, TABLE
+
+ assert {name: TABLE[name].cert_arch for name in _PTX_95_SYNC_FLOORS} ==
_PTX_95_SYNC_FLOORS
+ assert set(_PTX_95_SYNC_FLOORS) <= {entry.name for entry in
_PTX_95_ENTRIES}
+ # `.read` and `.sync_restrict::proxy::async` are separate syntax lines.
+ proxy = TABLE["cp_async_bulk_wait_group_proxy_async"]
+ assert "read" not in {slot.name for slot in proxy.slots}
+
+
+def _ptx_95_sync_kernel():
+ @T.function
+ def kernel(A: T.Tensor((32,), "uint32")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=32))
+ tx = T.cuda.thread_idx("x")
+ T.ptx.fabric.submit.commit_group()
+ T.ptx.fabric.wait.sync_restrict__read(0)
+ T.ptx["fabric.wait.sync_restrict::read"](3)
+ T.ptx.fabric.wait.sync_restrict__reads()
+ T.ptx.cp.async_.bulk.wait_group.sync_restrict__proxy__async(2)
+ T.ptx.cp.async_.bulk.wait_group(1)
+ T.ptx.cp.async_.bulk.wait_group.read(0)
+ T.ptx.fence.alias.sync_restrict__read.release.sys()
+ A[tx] = tx + T.uint32(7)
+
+ return kernel
+
+
[email protected](
+ not env.has_nvcc_version(13, 5),
+ reason="fabric commit groups, proxy-restricted bulk waits and the alias
fence are PTX 9.5",
+)
+@requires_nvcc
+def test_ptx_95_sync_codegen():
+ """The PTX 9.5 lines render verbatim beside their pre-9.5 siblings and
assemble."""
+ src = _cuda_source(
+ _ptx_95_sync_kernel(), tvm_tirx.target.Target({"kind": "cuda", "arch":
"sm_100a"})
+ )
+ for text in (
+ "fabric.submit.commit_group;",
+ "fabric.wait.sync_restrict::read 0;",
+ "fabric.wait.sync_restrict::read 3;",
+ "fabric.wait.sync_restrict::reads;",
+ "cp.async.bulk.wait_group.sync_restrict::proxy::async 2;",
+ "cp.async.bulk.wait_group 1;",
+ "cp.async.bulk.wait_group.read 0;",
+ "fence.alias.sync_restrict::read.release.sys;",
+ ):
+ assert text in src, text
+ # Every one of them orders memory without naming an address.
+ assert src.count('"fabric.submit.commit_group;" : : : "memory"') == 1
+ _assert_ptxas_ok(src, arch="sm_100a")
+
+
[email protected]
[email protected](
+ not env.has_nvcc_version(13, 5),
+ reason="fabric commit groups, proxy-restricted bulk waits and the alias
fence are PTX 9.5",
+)
[email protected](
+ not env.has_cuda() or env.cuda_arch() is None or int(env.cuda_arch()[3:6])
< 100,
+ reason="needs an sm_100+ CUDA GPU",
+)
+def test_ptx_95_sync_gpu_executes():
+ """With nothing pending, every PTX 9.5 wait returns at once and the kernel
completes."""
+ mod = tvm_tirx.compile(
+ tvm_tirx.IRModule({"main": _ptx_95_sync_kernel()}), target=TARGET,
pipeline="tirx"
+ )
+
+ def run_and_check():
+ dev = tvm_tirx.cuda(0)
+ a = tvm_tirx.runtime.tensor(np.zeros(32, dtype=np.uint32), device=dev)
+ mod(a)
+ np.testing.assert_array_equal(a.numpy(), np.arange(32,
dtype=np.uint32) + 7)
+
+ tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+
+
+def test_ptx_95_sin_cos_2pi_variants():
+ """sin/cos gain PTX 9.5's `.2pi`, legal only together with `.ftz`; lg2/ex2
do not."""
+ from tvm_tirx.backend.cuda.ptx.render import render_variant
+ from tvm_tirx.backend.cuda.ptx.table import TABLE, tokens_for, variants
+
+ for name in ("sin", "cos"):
+ entry = TABLE[name]
+ assert entry.cert_arch is None # "Supported on all target
architectures."
+ assert entry.check({"mode": "approx", "unit": "2pi", "ftz": "",
"type": "f32"})
+ assert len(variants(entry)) == 3 # {plain, .ftz, .2pi.ftz}
+ opcode = render_variant(
+ entry, tokens_for(entry, mode="approx", unit="2pi", ftz="ftz",
type="f32")
+ )[0]
+ assert opcode == f"{name}.approx.2pi.ftz.f32"
+ for name in ("lg2", "ex2"):
+ assert "unit" not in {slot.name for slot in TABLE[name].slots}
+
+
+def _ptx_95_sin_cos_2pi_kernel():
+ @T.function
+ def kernel(
+ A: T.Tensor((256,), "float32"),
+ S: T.Tensor((256,), "float32"),
+ C: T.Tensor((256,), "float32"),
+ ):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=256))
+ tx = T.cuda.thread_idx("x")
+ s = T.local_scalar("float32")
+ c = T.local_scalar("float32")
+ T.ptx["sin.approx.2pi.ftz.f32"](s, A[tx])
+ T.ptx["cos.approx.2pi.ftz.f32"](c, A[tx])
+ S[tx] = s
+ C[tx] = c
+
+ return kernel
+
+
[email protected](not env.has_nvcc_version(13, 5), reason="sin/cos .2pi is a
PTX 9.5 form")
+@requires_nvcc
+def test_ptx_95_sin_cos_2pi_codegen():
+ src = _cuda_source(_ptx_95_sin_cos_2pi_kernel())
+ assert "sin.approx.2pi.ftz.f32 %0, %1;" in src
+ assert "cos.approx.2pi.ftz.f32 %0, %1;" in src
+ _assert_ptxas_ok(src)
+
+
[email protected]
[email protected](not env.has_nvcc_version(13, 5), reason="sin/cos .2pi is a
PTX 9.5 form")
[email protected](not env.has_cuda(), reason="CUDA GPU not available")
+def test_ptx_95_sin_cos_2pi_gpu_accuracy():
+ """`.2pi` reads its input in cycles; check the ISA's stated absolute error
bounds.
+
+ 9.7.3.18/19: for cycles as input the maximum absolute error is 2^-20 (sin)
and
+ 2^-19.9 (cos) over [-1, 1], and 2^-14.2 for both over [-50, 50].
+ """
+ mod = tvm_tirx.compile(
+ tvm_tirx.IRModule({"main": _ptx_95_sin_cos_2pi_kernel()}),
target=TARGET, pipeline="tirx"
+ )
+ rng = np.random.default_rng(0)
+ cases = (
+ (np.float32(1.0), 2.0**-20, 2.0**-19.9),
+ (np.float32(50.0), 2.0**-14.2, 2.0**-14.2),
+ )
+
+ def run_and_check():
+ dev = tvm_tirx.cuda(0)
+ for bound, sin_tol, cos_tol in cases:
+ x = rng.uniform(-bound, bound, 256).astype(np.float32)
+ # Exact turns and quarter turns, where the result is exactly 0 or
+-1.
+ x[:9] = np.array([0, 0.25, 0.5, 0.75, 1, -0.25, -0.5, -0.75, -1],
dtype=np.float32)
+ a = tvm_tirx.runtime.tensor(x, device=dev)
+ s = tvm_tirx.runtime.tensor(np.zeros(256, dtype=np.float32),
device=dev)
+ c = tvm_tirx.runtime.tensor(np.zeros(256, dtype=np.float32),
device=dev)
+ mod(a, s, c)
+ turns = 2.0 * np.pi * x.astype(np.float64)
+ assert np.max(np.abs(s.numpy() - np.sin(turns))) <= sin_tol
+ assert np.max(np.abs(c.numpy() - np.cos(turns))) <= cos_tol
+
+ tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+
+
+_PTX_95_TENSORMAP_ENTRIES = (
+ "tensormap_replace_address_alloc_class",
+ "tensormap_replace_alloc_class",
+ "tensormap_query",
+)
+
+
+def test_ptx_95_tensormap_alloc_class_entries():
+ """The 9.5 allocation-class lines certify at sm_90a; `.alloc_same` needs a
field."""
+ from tvm_tirx.backend.cuda.ptx.table import _PTX_95_ENTRIES, TABLE
+
+ delta = {entry.name for entry in _PTX_95_ENTRIES}
+ for name in _PTX_95_TENSORMAP_ENTRIES:
+ assert name in delta and TABLE[name].cert_arch == "sm_90a"
+
+ def alloc_choices(name):
+ (slot,) = [s for s in TABLE[name].slots if s.name == "alloc_class"]
+ return slot.choices
+
+ assert alloc_choices("tensormap_replace_address_alloc_class") == (
+ "alloc_small",
+ "alloc_large",
+ "alloc_same",
+ )
+ assert alloc_choices("tensormap_replace_alloc_class") == ("alloc_small",
"alloc_large")
+
+
+def _ptx_95_tensormap_alloc_class_kernel():
+ @T.function
+ def kernel(TM: T.Tensor((32,), "uint32"), OUT: T.Tensor((6,), "uint32")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=1))
+ p = T.local_scalar("uint32")
+ smem = T.alloc_tensor(ty_args=[T.Tensor((32,), "uint32",
scope="shared")])
+ tmap = TM.ptr_to([0])
+ query = T.ptx["tensormap_query.alloc_class_large.global.b1024"]
+ # Field-less lines.
+ T.ptx["tensormap_replace.tile.alloc_large.global.b1024"](tmap)
+ query(p, tmap)
+ OUT[0] = p
+ T.ptx["tensormap_replace.tile.alloc_small.global.b1024"](tmap)
+ query(p, tmap)
+ OUT[1] = p
+ # `.global_address` with an explicit class.
+
T.ptx["tensormap_replace.tile.global_address.alloc_large.global.b1024.b64"](
+ tmap, T.uint64(0x10000)
+ )
+ query(p, tmap)
+ OUT[2] = p
+ # PTX 9.5 clarification: `.global_address` without `.alloc_class` sets
Small.
+ T.ptx["tensormap_replace.tile.global_address.global.b1024.b64"](tmap,
T.uint64(0x20000))
+ query(p, tmap)
+ OUT[3] = p
+ # `.alloc_same` keeps whatever class the object already has.
+ T.ptx["tensormap_replace.tile.alloc_large.global.b1024"](tmap)
+
T.ptx["tensormap_replace.tile.global_address.alloc_same.global.b1024.b64"](
+ tmap, T.uint64(0x30000)
+ )
+ query(p, tmap)
+ OUT[4] = p
+ # Generic addressing reads the same object.
+ T.ptx["tensormap_query.alloc_class_large.b1024"](p, tmap)
+ OUT[5] = p
+ # The shared::cta replaces assemble and run; the query has no
shared::cta
+ # helper (see the table), so their effect is not observed here.
+
T.ptx["tensormap_replace.tile.alloc_small.shared::cta.b1024"](smem.ptr_to([0]))
+
T.ptx["tensormap_replace.tile.global_address.alloc_large.shared::cta.b1024.b64"](
+ smem.ptr_to([0]), T.uint64(0x40000)
+ )
+
+ return kernel
+
+
[email protected](
+ not env.has_nvcc_version(13, 5), reason="tensor-map allocation classes are
PTX 9.5"
+)
+@requires_nvcc
+def test_ptx_95_tensormap_alloc_class_codegen():
+ src = _cuda_source(
+ _ptx_95_tensormap_alloc_class_kernel(),
+ tvm_tirx.target.Target({"kind": "cuda", "arch": "sm_90a"}),
+ )
+ for text in (
+ "tensormap.replace.tile.alloc_large.global.b1024 [%0];",
+ "tensormap.replace.tile.alloc_small.shared::cta.b1024 [%0];",
+
"tensormap.replace.tile.global_address.alloc_large.shared::cta.b1024.b64 [%0],
%1;",
+ "tensormap.replace.tile.global_address.alloc_large.global.b1024.b64
[%0], %1;",
+ "tensormap.replace.tile.global_address.alloc_same.global.b1024.b64
[%0], %1;",
+ "tensormap.query.alloc_class_large.global.b1024 pd0, [%1];",
+ "tensormap.query.alloc_class_large.b1024 pd0, [%1];",
+ ):
+ assert text in src, text
+ _assert_ptxas_ok(src, arch="sm_90a")
+
+
[email protected]
[email protected](
+ not env.has_nvcc_version(13, 5), reason="tensor-map allocation classes are
PTX 9.5"
+)
[email protected](
+ not env.has_cuda() or env.cuda_arch() is None or int(env.cuda_arch()[3:6])
< 90,
+ reason="needs an sm_90+ CUDA GPU",
+)
+def test_ptx_95_tensormap_alloc_class_gpu():
+ """Each replace leaves the class the ISA says, as observed by the query."""
+ mod = tvm_tirx.compile(
+ tvm_tirx.IRModule({"main": _ptx_95_tensormap_alloc_class_kernel()}),
+ target=TARGET,
+ pipeline="tirx",
+ )
+
+ def run_and_check():
+ dev = tvm_tirx.cuda(0)
+ tm = tvm_tirx.runtime.tensor(np.zeros(32, dtype=np.uint32), device=dev)
+ out = tvm_tirx.runtime.tensor(np.full(6, 7, dtype=np.uint32),
device=dev)
+ mod(tm, out)
+ np.testing.assert_array_equal(out.numpy(), [1, 0, 1, 0, 1, 1])
+
+ tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+
+
+_SM107F_PACKED_INT_OPCODES = {
+ "add_int_sm107f": {
+ "add.u8x4",
+ "add.s8x4",
+ "add.sat.u8x4",
+ "add.sat.s8x4",
+ "add.sat.u32",
+ "add.sat.u16x2",
+ "add.sat.s16x2",
+ },
+ "sub_int_sm107f": {"sub.u8x4", "sub.s8x4", "sub.sat.u8x4", "sub.sat.s8x4"},
+ "neg_int_sm107f": {"neg.s8x4"},
+ "min_int8x4": {"min.u8x4", "min.s8x4", "min.relu.s8x4"},
+ "max_int8x4": {"max.u8x4", "max.s8x4", "max.relu.s8x4"},
+}
+
+
+def test_ptx_sm107f_packed_int_entries():
+ """The sm_107f/sm_120f packed-integer lines are sm_107f siblings, exactly
these opcodes."""
+ from tvm_tirx.backend.cuda.ptx.render import render_variant
+ from tvm_tirx.backend.cuda.ptx.table import TABLE, variants
+
+ for name, expected in _SM107F_PACKED_INT_OPCODES.items():
+ entry = TABLE[name]
+ assert entry.cert_arch == "sm_107f", name
+ assert {render_variant(entry, tokens)[0] for tokens in
variants(entry)} == expected
+ # The families they extend keep their pre-sm_107 floors.
+ assert TABLE["add_int"].cert_arch == "sm_90"
+ assert TABLE["sub_int"].cert_arch is None and TABLE["neg_int"].cert_arch
is None
+
+
+def _sm107f_packed_int_kernel():
+ @T.function
+ def kernel(
+ A: T.Tensor((256,), "uint32"),
+ B: T.Tensor((256,), "uint32"),
+ D: T.Tensor((16, 256), "uint32"),
+ ):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=256))
+ tx = T.cuda.thread_idx("x")
+ a = A[tx]
+ b = B[tx]
+ d = T.local_scalar("uint32")
+ T.ptx.add.u8x4(d, a, b)
+ D[0, tx] = d
+ T.ptx.add.sat.u8x4(d, a, b)
+ D[1, tx] = d
+ T.ptx.add.s8x4(d, a, b)
+ D[2, tx] = d
+ T.ptx.add.sat.s8x4(d, a, b)
+ D[3, tx] = d
+ T.ptx.sub.u8x4(d, a, b)
+ D[4, tx] = d
+ T.ptx.sub.sat.u8x4(d, a, b)
+ D[5, tx] = d
+ T.ptx.sub.sat.s8x4(d, a, b)
+ D[6, tx] = d
+ T.ptx.neg.s8x4(d, a)
+ D[7, tx] = d
+ T.ptx.min.u8x4(d, a, b)
+ D[8, tx] = d
+ T.ptx.max.s8x4(d, a, b)
+ D[9, tx] = d
+ T.ptx.min.relu.s8x4(d, a, b)
+ D[10, tx] = d
+ T.ptx.max.relu.s8x4(d, a, b)
+ D[11, tx] = d
+ T.ptx.add.sat.u32(d, a, b)
+ D[12, tx] = d
+ T.ptx.add.sat.u16x2(d, a, b)
+ D[13, tx] = d
+ T.ptx.add.sat.s16x2(d, a, b)
+ D[14, tx] = d
+ T.ptx.add.u32(d, a, b) # the sm_90 `add_int` line, still selected
+ D[15, tx] = d
+
+ return kernel
+
+
[email protected](not env.has_nvcc_version(13, 4), reason="need nvcc >= 13.4
for sm_107f")
+@requires_nvcc
+def test_ptx_sm107f_packed_int_codegen():
+ src = _cuda_source(
+ _sm107f_packed_int_kernel(), tvm_tirx.target.Target({"kind": "cuda",
"arch": "sm_107f"})
+ )
+ for opcode in (
+ "add.u8x4",
+ "add.sat.u8x4",
+ "add.s8x4",
+ "add.sat.s8x4",
+ "sub.u8x4",
+ "sub.sat.u8x4",
+ "sub.sat.s8x4",
+ "neg.s8x4",
+ "min.u8x4",
+ "max.s8x4",
+ "min.relu.s8x4",
+ "max.relu.s8x4",
+ "add.sat.u32",
+ "add.sat.u16x2",
+ "add.sat.s16x2",
+ "add.u32",
+ ):
+ assert f'"{opcode} %0, %1' in src, opcode
+ _assert_ptxas_ok(src, arch="sm_107f")
+
+
+def _lanes(x, dtype, n):
+ return x.view(dtype).reshape(-1, n).astype(np.int64)
+
+
+def _pack(lanes, dtype):
+ info = np.iinfo(dtype)
+ wrapped = (lanes - info.min) % (info.max - info.min + 1) + info.min
+ return wrapped.astype(dtype).reshape(-1).view(np.uint32)
+
+
[email protected]
[email protected](not env.has_nvcc_version(13, 4), reason="need nvcc >= 13.4
for sm_107f")
[email protected](
+ env.cuda_arch() not in ("sm_107a", "sm_120a", "sm_121a"),
+ reason="needs an sm_107/sm_12x GPU (the packed int lines are
sm_107f/sm_120f-only)",
+)
+def test_ptx_sm107f_packed_int_gpu():
+ """Lane-wise wrap-around and saturation semantics against numpy."""
+ mod = tvm_tirx.compile(
+ tvm_tirx.IRModule({"main": _sm107f_packed_int_kernel()}),
target=TARGET, pipeline="tirx"
+ )
+ rng = np.random.default_rng(0)
+ a = rng.integers(0, 2**32, 256, dtype=np.uint64).astype(np.uint32)
+ b = rng.integers(0, 2**32, 256, dtype=np.uint64).astype(np.uint32)
+ # Edge lanes: overflow/underflow in every lane, and the most negative
value.
+ a[:4] = [0xFFFFFFFF, 0x80808080, 0x7F7F7F7F, 0x00000000]
+ b[:4] = [0x01010101, 0x80808080, 0x01010101, 0xFFFFFFFF]
+
+ def clamp(x, dtype):
+ info = np.iinfo(dtype)
+ return np.clip(x, info.min, info.max)
+
+ u8a, u8b = _lanes(a, np.uint8, 4), _lanes(b, np.uint8, 4)
+ s8a, s8b = _lanes(a, np.int8, 4), _lanes(b, np.int8, 4)
+ u16a, u16b = _lanes(a, np.uint16, 2), _lanes(b, np.uint16, 2)
+ s16a, s16b = _lanes(a, np.int16, 2), _lanes(b, np.int16, 2)
+ a64, b64 = a.astype(np.int64), b.astype(np.int64)
+ expected = [
+ _pack(u8a + u8b, np.uint8),
+ _pack(clamp(u8a + u8b, np.uint8), np.uint8),
+ _pack(s8a + s8b, np.int8),
+ _pack(clamp(s8a + s8b, np.int8), np.int8),
+ _pack(u8a - u8b, np.uint8),
+ _pack(clamp(u8a - u8b, np.uint8), np.uint8),
+ _pack(clamp(s8a - s8b, np.int8), np.int8),
+ _pack(-s8a, np.int8),
+ _pack(np.minimum(u8a, u8b), np.uint8),
+ _pack(np.maximum(s8a, s8b), np.int8),
+ _pack(np.maximum(np.minimum(s8a, s8b), 0), np.int8),
+ _pack(np.maximum(np.maximum(s8a, s8b), 0), np.int8),
+ np.minimum(a64 + b64, 2**32 - 1).astype(np.uint32),
+ _pack(clamp(u16a + u16b, np.uint16), np.uint16),
+ _pack(clamp(s16a + s16b, np.int16), np.int16),
+ ((a64 + b64) % 2**32).astype(np.uint32),
+ ]
+
+ def run_and_check():
+ dev = tvm_tirx.cuda(0)
+ d = tvm_tirx.runtime.tensor(np.zeros((16, 256), dtype=np.uint32),
device=dev)
+ mod(tvm_tirx.runtime.tensor(a, device=dev), tvm_tirx.runtime.tensor(b,
device=dev), d)
+ got = d.numpy()
+ for row, want in enumerate(expected):
+ np.testing.assert_array_equal(got[row], want, err_msg=f"row {row}")
+
+ tvm_tirx.testing.run_with_gpu_lock(run_and_check)
+
+
+@requires_nvcc
[email protected](not env.has_nvcc_version(13, 4), reason="need nvcc >= 13.4
for sm_107a")
+def test_ptx_tcgen05_shift_codegen():
+ """tcgen05.shift takes a bracketed tmem address and exists only on `a`
targets."""
+ from tvm_tirx.backend.cuda.ptx.render import render_variant
+ from tvm_tirx.backend.cuda.ptx.table import TABLE, variants
+
+ entry = TABLE["tcgen05_shift"]
+ assert entry.cert_arch == "sm_100a"
+ assert {render_variant(entry, tokens)[0] for tokens in variants(entry)} ==
{
+ "tcgen05.shift.cta_group::1.down",
+ "tcgen05.shift.cta_group::2.down",
+ }
+
+ @T.function
+ def kernel(A: T.Tensor((32,), "uint32")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=32))
+ tx = T.cuda.thread_idx("x")
+ if tx == 0:
+ T.ptx.tcgen05.shift.cta_group__1.down(A[0])
+ A[tx] = A[tx]
+
+ for arch in ("sm_100a", "sm_107a"):
+ src = _cuda_source(kernel, tvm_tirx.target.Target({"kind": "cuda",
"arch": arch}))
+ assert "tcgen05.shift.cta_group::1.down [%0];" in src
+ _assert_ptxas_ok(src, arch=arch)
+
+
+def test_ptx_every_modifier_token_is_spellable():
+ """No modifier choice may contain a dot: `T.ptx[...]` and attribute chains
split on it."""
+ from tvm_tirx.backend.cuda.ptx.table import TABLE
+
+ dotted = sorted(
+ (entry.name, slot.name, choice)
+ for entry in TABLE.values()
+ for slot in entry.slots
+ for choice in slot.choices
+ if "." in choice
+ )
+ assert not dotted, dotted
+
+
[email protected](not env.has_nvcc_version(13, 3), reason="fabric operations
are PTX 9.3")
+@requires_nvcc
+def test_ptx_fabric_called_through_string_form():
+ """The PTX 9.3 fabric operations are callable by their exact PTX text."""
+
+ @T.function
+ def kernel(A: T.Tensor((64,), "uint32"), P: T.Tensor((8,), "uint64")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=32))
+ tx = T.cuda.thread_idx("x")
+ smem = T.alloc_tensor(ty_args=[T.Tensor((64,), "uint32",
scope="shared")])
+ mbar = T.alloc_tensor(ty_args=[T.Tensor((2,), "uint64",
scope="shared")])
+ if tx == 0:
+ T.ptx[
+ "fabric.try_get.async.shared::cta"
+
".mbarrier::complete_tx::bytes.mbarrier::report::fabric.relaxed.sys.b128"
+ ](smem.ptr_to([0]), A[0], P[1], T.uint32(16), mbar.ptr_to([0]))
+ T.ptx[
+ "fabric.try_put.async.shared::cta.mbarrier::complete_tx::16B"
+ ".mbarrier::report::fabric.counted::bytes.relaxed.sys.b128"
+ ](A[0], P[1], P[2], smem.ptr_to([0]), T.uint32(16),
mbar.ptr_to([0]))
+ T.ptx.fabric.submit()
+ A[tx] = A[tx]
+
+ src = _cuda_source(kernel, tvm_tirx.target.Target({"kind": "cuda", "arch":
"sm_103a"}))
+ assert (
+
"fabric.try_get.async.shared::cta.mbarrier::complete_tx::bytes.mbarrier::report::fabric"
+ ".relaxed.sys.b128 [%0], [%1, %2], %3, [%4];" in src
+ )
+ assert ".counted::bytes.relaxed.sys.b128 [%0, %1, %2], [%3], %4, [%5];" in
src
+ _assert_ptxas_ok(src, arch="sm_103a")
+
+
+def test_ptx_fabric_tensor_and_atom_entries():
+ """PTX 9.4's fabric tensor operations and try_atom, in the 9.4 delta at
their floors."""
+ from tvm_tirx.backend.cuda.ptx.render import render_variant
+ from tvm_tirx.backend.cuda.ptx.table import _PTX_94_ENTRIES, TABLE,
tokens_for, variants
+
+ delta = {entry.name for entry in _PTX_94_ENTRIES}
+ tensor = {name for name in TABLE if name.startswith("fabric_try_") and
"_tensor" in name}
+ assert len(tensor) == 25 # get + put/red x {plain, counted}, x 5 override
shapes
+ assert tensor <= delta and {TABLE[name].cert_arch for name in tensor} ==
{"sm_107f"}
+ assert "fabric_try_atom" in delta and TABLE["fabric_try_atom"].cert_arch
== "sm_100"
+
+ atom = TABLE["fabric_try_atom"]
+ pairs = {
+ tuple(render_variant(atom, tokens)[0].rsplit(".", 2)[-2:]) for tokens
in variants(atom)
+ }
+ assert {("add", "f16x2"), ("cas", "b128")} <= pairs
+ # Off the ISA grid; ptxas accepts `.min.s32`, but the table follows the
ISA.
+ assert not {("min", "f32"), ("add", "b32"), ("min", "s32")} & pairs
+ assert len(pairs) == 26 # 3x2 bitwise + 2x3 exch/cas + 2x4 min/max + 6 add
+
+ red = TABLE["fabric_try_red_tensor"]
+ opcode = render_variant(
+ red,
+ tokens_for(
+ red,
+ action="try_red",
+ api="async",
+ unit="tensor",
+ dim="2d",
+ src="shared::cta",
+ load_mode="tile",
+ completion="mbarrier::complete_tx::16B",
+ report="mbarrier::report::fabric",
+ override_address="override::global_address",
+ sem="relaxed",
+ scope="sys",
+ redop="max",
+ ),
+ )[0]
+ assert opcode.endswith(".override::global_address.relaxed.sys.max") # no
element type
+
+
[email protected](not env.has_nvcc_version(13, 4), reason="fabric tensor
operations are PTX 9.4")
+@requires_nvcc
+def test_ptx_fabric_tensor_codegen():
+ """Each override shape renders its handle bracket and assembles at
sm_107f."""
+
+ @T.function
+ def kernel(A: T.Tensor((64,), "uint32"), P: T.Tensor((8,), "uint64")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=32))
+ tx = T.cuda.thread_idx("x")
+ smem = T.alloc_tensor(ty_args=[T.Tensor((64,), "uint32",
scope="shared")])
+ mbar = T.alloc_tensor(ty_args=[T.Tensor((2,), "uint64",
scope="shared")])
+ if tx == 0:
+ tmap = P.ptr_to([0])
+ le = A[0]
+ off = P[1]
+ c0 = T.int32(A[1])
+ c1 = T.int32(A[2])
+ T.ptx[
+ "fabric.try_get.async.tensor.2d.shared::cta.tile"
+ ".mbarrier::complete_tx::bytes.mbarrier::report::fabric"
+ ".override::global_address.relaxed.sys.b128"
+ ](smem.ptr_to([0]), tmap, le, off, c0, c1, mbar.ptr_to([0]))
+ T.ptx[
+ "fabric.try_get.async.tensor.1d.shared::cta.tile"
+
".mbarrier::complete_tx::bytes.mbarrier::report::fabric.L2::cache_hint"
+
".override::global_address.override::global_dim.relaxed.sys.b128"
+ ](smem.ptr_to([0]), tmap, le, off, T.uint16(64), c0,
mbar.ptr_to([0]), P[2])
+ T.ptx[
+ "fabric.try_put.async.tensor.multimem.3d.shared::cta.tile"
+
".mbarrier::complete_tx::16B.mbarrier::report::fabric.counted::bytes"
+
".override::global_address.override::global_dim_stride.relaxed.sys.b128"
+ ](
+ tmap, le, off, P[3],
+ T.uint16(8), T.uint16(8), T.uint16(8),
+ T.uint32(256), T.uint32(2048),
+ T.uint16(0),
+ c0, c1, c0,
+ smem.ptr_to([0]), T.uint32(256), mbar.ptr_to([0]),
+ ) # fmt: skip
+ T.ptx[
+ "fabric.try_red.async.tensor.2d.shared::cta.tile"
+ ".mbarrier::complete_tx::16B.mbarrier::report::fabric"
+ ".override::global_address.relaxed.sys.add"
+ ](tmap, le, off, c0, c1, smem.ptr_to([0]), T.uint32(256),
mbar.ptr_to([0]))
+ A[tx] = A[tx]
+
+ src = _cuda_source(kernel, tvm_tirx.target.Target({"kind": "cuda", "arch":
"sm_107f"}))
+ for text in (
+ ".override::global_address.relaxed.sys.b128 [%0], [%1, %2, %3, {%4,
%5}], [%6];",
+ ".override::global_dim.relaxed.sys.b128 [%0], [%1, %2, %3, {%4},
{%5}], [%6], %7;",
+
".counted::bytes.override::global_address.override::global_dim_stride.relaxed.sys.b128
"
+ "[%0, %1, %2, %3, {%4, %5, %6}, {%7, %8}, %9, {%10, %11, %12}], [%13],
%14, [%15];",
+ ".override::global_address.relaxed.sys.add [%0, %1, %2, {%3, %4}],
[%5], %6, [%7];",
+ ):
+ assert text in src, text
+ _assert_ptxas_ok(src, arch="sm_107f")
+
+
+@requires_nvcc
[email protected](not env.has_nvcc_version(13, 4), reason="fabric.try_atom
is PTX 9.4")
+def test_ptx_fabric_try_atom_codegen():
+ @T.function
+ def kernel(A: T.Tensor((64,), "uint32"), P: T.Tensor((8,), "uint64")):
+ T.device_entry(launch=T.cuda.LaunchConfig(grid=1, block=32))
+ tx = T.cuda.thread_idx("x")
+ smem = T.alloc_tensor(ty_args=[T.Tensor((64,), "uint32",
scope="shared")])
+ mbar = T.alloc_tensor(ty_args=[T.Tensor((2,), "uint64",
scope="shared")])
+ if tx == 0:
+ T.ptx[
+ "fabric.try_atom.async.shared::cta"
+
".mbarrier::complete_tx::16B.mbarrier::report::fabric.relaxed.sys.add.bf16x2"
+ ](A[0], P[1], smem.ptr_to([0]), smem.ptr_to([16]),
mbar.ptr_to([0]))
+ T.ptx[
+ "fabric.try_atom.async.shared::cta"
+
".mbarrier::complete_tx::16B.mbarrier::report::fabric.L2::cache_hint"
+ ".relaxed.sys.cas.b128"
+ ](A[0], P[1], smem.ptr_to([0]), smem.ptr_to([16]),
mbar.ptr_to([0]), P[2])
+ A[tx] = A[tx]
+
+ src = _cuda_source(kernel, tvm_tirx.target.Target({"kind": "cuda", "arch":
"sm_100a"}))
+ assert ".relaxed.sys.add.bf16x2 [%0, %1], [%2], [%3], [%4];" in src
+ assert ".L2::cache_hint.relaxed.sys.cas.b128 [%0, %1], [%2], [%3], [%4],
%5;" in src
+ _assert_ptxas_ok(src, arch="sm_100a")
+
+
def test_ptx_tcgen05_mapa_address_rendering():
"""Optional state spaces select generic versus shared address carriers."""
from tvm_tirx.backend.cuda.ptx.render import render_variant
@@ -4159,7 +4823,7 @@ def test_ptx_all_variants_render_unique():
_, helper, _ = render_variant(entry, *args,
addr_offsets=addr_offsets)
assert helper not in names, f"address-offset helper name
collision: {helper}"
names.add(helper)
- assert total == 762178 # update when the table grows or a ptxas gap
narrows it
+ assert total == 791291 # update when the table grows or a ptxas gap
narrows it
def test_ptx_no_instruction_registered_twice():
@@ -4373,14 +5037,20 @@ _PTX_93_SM103A_PHASE_ENTRIES = frozenset(
def _ptx_93_sm103a_manifest():
"""Yield every rendering introduced or extended by PTX 9.3 for SM103a."""
- from tvm_tirx.backend.cuda.ptx.table import _PTX_94_ENTRIES, TABLE, mods,
renderings
+ from tvm_tirx.backend.cuda.ptx.table import (
+ _PTX_94_ENTRIES,
+ _PTX_95_ENTRIES,
+ TABLE,
+ mods,
+ renderings,
+ )
- ptx94 = {entry.name for entry in _PTX_94_ENTRIES}
+ later = {entry.name for entry in (*_PTX_94_ENTRIES, *_PTX_95_ENTRIES)}
for name, entry in TABLE.items():
- if name in ptx94:
- # The 9.4 delta (sm_107f floors, including the cp.async.bulk
- # `*_report` entries) is certified at its own arch by
- # test_ptx_all_helpers_certify, not at sm_103a here.
+ if name in later:
+ # The 9.4 and 9.5 deltas (sm_107f floors, the cp.async.bulk
+ # `*_report` entries, fabric submit groups) are certified at their
+ # own arch by test_ptx_all_helpers_certify, not at sm_103a here.
continue
for rendering in renderings(entry):
tokens = rendering[0]