Issue 243218
Summary [mlir][tosa] `TransposeOp::reifyResultShapes` crashes on unranked tensor input
Labels mlir
Assignees
Reporter quiet-maple-47
    ## Description

`--resolve-shaped-type-result-dims` crashes (or exhausts memory / fails `SmallVector` allocation) when `tosa.transpose` takes an unranked tensor (`tensor<*xi32>`) as input.

`tosa::TransposeOp::reifyResultShapes` calls `inputType.getRank()` without checking `inputType.hasRank()`, allocating a `SmallVector` with `kDynamic` (`INT64_MIN`) elements on non-assertions builds.

## Reproducer

`input.mlir`:

```mlir
func.func @f(%arg0: tensor<*xi32>) -> index {
  %c0 = arith.constant 0 : index
  %0 = tosa.transpose %arg0 perms([0, 2, 1, 3]) : (tensor<*xi32>) -> tensor<?x3x2x2xi32>
  %d = tensor.dim %0, %c0 : tensor<?x3x2x2xi32>
  return %d : index
}
```

```bash
mlir-opt --resolve-shaped-type-result-dims input.mlir
```

[Reproduce on Compiler Explorer (LLVM trunk)](https://godbolt.org/z/5zdPs7Md8)

## Actual Behavior

On LLVM trunk (Compiler Explorer), `mlir-opt` is terminated with `SIGKILL` (exit code 137) while trying to allocate `INT64_MIN` elements for `SmallVector<OpFoldResult>`.

On Homebrew LLVM 23.1.5 (with `{perms = ...}` syntax), `mlir-opt` aborts with:

```text
LLVM ERROR: SmallVector unable to grow. Requested capacity (505012822184) is larger than maximum value for size type (4294967295)
```

On assertions-enabled builds (LLVM 23 `b3df0665`), `mlir-opt` aborts in `ShapedTypeTrait<TensorType>::getRank()`:

```text
mlir-opt: mlir/include/mlir/IR/BuiltinTypeInterfaces.h.inc:758: int64_t mlir::detail::ShapedTypeTrait<mlir::TensorType>::getRank() const [ConcreteType = mlir::TensorType]: Assertion `(*static_cast<const ConcreteType *>(this)).hasRank() && "cannot query rank of unranked shaped type"' failed.
```

## Expected Behavior

`TransposeOp::reifyResultShapes` should return `failure()` when `inputType` is unranked (or derive rank/dimensions safely when unranked), rather than querying `getRank()` on an `UnrankedTensorType`.

## Root Cause

[`tosa::TransposeOp::reifyResultShapes`](https://github.com/llvm/llvm-project/blob/e0c9350a3f88ca5a37aaac4c7cc9caebf5c309e0/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp#L3267-L3289) casts `input.getType()` to `TensorType` and immediately constructs `SmallVector<OpFoldResult> returnedDims(inputType.getRank())` at line 3275 without checking `inputType.hasRank()`.

Checking `if (!inputType.hasRank()) return failure();` (or using `RankedTensorType` / `transposePerms.size()`) prevents the crash.

## Version

- LLVM trunk on Compiler Explorer (`mliropttrunk`, LLVM 24.0.0git)
- Homebrew LLVM 23.1.5 (macOS arm64; using `{perms = ...}` syntax)
- `llvm-project` `b3df0665` (Linux x86_64, assertions enabled; using `{perms = ...}` syntax)
- Relevant code verified on `main` at `e0c9350a3f88` (2026-11-16)

## Claim

This investigation and write-up were AI-assisted. I've reviewed and verified everything above, and I'm happy to answer questions.

**Assisted-by:** Gemini / Antigravity

_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs

Reply via email to