This is an automated email from the ASF dual-hosted git repository.
tlopex pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new 70b2640066 [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss
from hardcoded threshold (#20212)
70b2640066 is described below
commit 70b264006695562893afc954e95c0278ce0a835a
Author: HuEnwei <[email protected]>
AuthorDate: Fri Aug 28 16:06:06 2026 +0800
[Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded
threshold (#20212)
Fixes: #20184
## Summary
The Relax ONNX frontend `Softplus._impl_v1` passed a hardcoded
`threshold`
(`10.0` for float16, `20.0` otherwise) to `relax.op.nn.softplus`. ONNX
`Softplus` is defined as `y = log(exp(x) + 1)` with **no threshold**
attribute, but the topi lowering of `relax.op.nn.softplus` clamps the
result
to the linear function `x` once `beta * x` exceeds the threshold:
```python
# python/tvm/topi/nn/elemwise.py
return tvm.tir.Select(b * value > t, value, (1 / b) * tvm.tir.log(1 +
tvm.tir.exp(b * value)))
```
## Root cause
The `threshold` is chosen once for every dtype, and `20` is far too
small for
float64: `log(exp(x) + 1) = x + log(1 + exp(-x))` is fully representable
in
float64 up to `x ≈ 709`, so clamping at `x = 20` drops the
`log(1 + exp(-x))` term. Differential test vs `onnx.reference` showed an
observable absolute error of `1.87e-9` at `x = 20.1`:
| x (float64) | `onnx.reference` | TVM (before) |
|---|---|---|
| 20.0 | 20.000000002061153 | 20.000000002061153 |
| 20.1 | 20.10000000186501 | **20.1** (err 1.87e-9) |
| 25.0 | 25.000000000013888 | **25.0** (err 1.39e-11) |
(For float32/float16 the clamp is numerically invisible in ordinary
ranges —
`log(exp(x)+1)` already rounds to `x` for `x ≳ 15` — and near overflow
both
onnxruntime and TVM intentionally return a finite value, so the
observable,
spec-relevant defect is the float64 case.)
## Fix
Lower ONNX `Softplus` as the numerically stable, threshold-free form:
```python
x = inputs[0]
dtype = x.ty.dtype
return relax.op.add(
relax.op.maximum(x, relax.const(0, dtype)),
relax.op.log(
relax.op.add(
relax.const(1, dtype),
relax.op.exp(relax.op.negative(relax.op.abs(x))),
)
),
)
```
`max(x, 0) + log(1 + exp(-|x|))` equals `log(exp(x) + 1)` mathematically
for
every finite `x`: it never overflows (both terms are bounded), it
matches
onnxruntime on float32/float16 (including near overflow, where the naive
formula would overflow to `inf`), and it keeps float64 exact across the
whole
representable range. The torch frontend keeps using
`relax.op.nn.softplus`
with `beta`/`threshold` since PyTorch's `nn.Softplus` genuinely has
those
parameters.
## Validation
Differential test: Relax (build + `VirtualMachine`) vs onnxruntime
(float32/float16) and the mathematically exact stable form cross-checked
with
`onnx.reference` (float64; onnxruntime's CPU EP has no float64
Softplus), across
3 dtypes × full value range (incl. the old failure region `x ∈ (20, 30]`
and
near-overflow) × multiple shapes and opsets.
| Case | Result |
|---|---|
| fp32 regular / threshold-region / near-overflow `[60..100]` | all OK
(vs onnxrt) |
| fp64 regular / `x ∈ (19..50]` (old failure region) / `[600..1000]` |
all OK, max\|diff\|=0 (vs exact) |
| fp16 regular / threshold-region `[10..12]` | all OK (vs onnxrt) |
| opset 1 / 13 / 22 (fp32) | all OK |
| fp64 `x = 20.1 / 25 / 30` (pre-fix failures) | max\|diff\| = 0.00e+00
|
Total: **19 differential cases, 19 OK, 0 rejected, 0 numeric
mismatches**.
In-tree tests `test_softplus_large_values` (float32 up to `x = 80` vs
onnxruntime) and `test_softplus_float64_accuracy` (float64 vs exact,
fails
before this fix) were added; the existing `test_unary[Softplus]` still
passes.
## Files changed
- `python/tvm/relax/frontend/onnx/onnx_frontend.py` —
`Softplus._impl_v1`:
lower numerically stable `max(x, 0) + log(1 + exp(-|x|))` instead of
`relax.op.nn.softplus` with a hardcoded `threshold`.
- `tests/python/relax/test_frontend_onnx.py` — add
`test_softplus_large_values` and `test_softplus_float64_accuracy`.
## Base
Based on current upstream `main` (`ad0a225074`, 2026-08-28) using the
current
`x.ty.dtype` API (vs the older `struct_info.dtype`). The single commit
`e5eede1b5d` applies cleanly on `main` with `git am`/`git apply`.
---
python/tvm/relax/frontend/onnx/onnx_frontend.py | 14 +++++--
tests/python/relax/test_frontend_onnx.py | 53 +++++++++++++++++++++++++
2 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/python/tvm/relax/frontend/onnx/onnx_frontend.py
b/python/tvm/relax/frontend/onnx/onnx_frontend.py
index 024b2bf2b6..0bbfb4e281 100644
--- a/python/tvm/relax/frontend/onnx/onnx_frontend.py
+++ b/python/tvm/relax/frontend/onnx/onnx_frontend.py
@@ -2662,9 +2662,17 @@ class Softplus(OnnxOpConverter):
@classmethod
def _impl_v1(cls, bb, inputs, attr, params):
- dtype = inputs[0].ty.dtype
- threshold = 10.0 if dtype == "float16" else 20.0
- return relax.op.nn.softplus(inputs[0], threshold=threshold)
+ x = inputs[0]
+ dtype = x.ty.dtype
+ return relax.op.add(
+ relax.op.maximum(x, relax.const(0, dtype)),
+ relax.op.log(
+ relax.op.add(
+ relax.const(1, dtype),
+ relax.op.exp(relax.op.negative(relax.op.abs(x))),
+ )
+ ),
+ )
class Softsign(OnnxOpConverter):
diff --git a/tests/python/relax/test_frontend_onnx.py
b/tests/python/relax/test_frontend_onnx.py
index 91b2e2ded7..984e05aff9 100644
--- a/tests/python/relax/test_frontend_onnx.py
+++ b/tests/python/relax/test_frontend_onnx.py
@@ -3973,6 +3973,59 @@ def test_prelu_multi_axis_slope():
check_correctness(model, inputs=inputs, opset=16, check_dtypes=True)
+def test_softplus_large_values():
+ """ONNX Softplus is y = log(exp(x) + 1) for every finite input. The
frontend
+ must not clamp the output to the linear function x beyond a hardcoded
threshold
+ (it used to pass threshold=20 to relax.op.nn.softplus). Large float32
inputs,
+ including values beyond that threshold, must still match onnxruntime."""
+ for shape in [[3, 32, 32], [5]]:
+ graph = helper.make_graph(
+ [helper.make_node("Softplus", ["x"], ["y"])],
+ "softplus_large_values",
+ inputs=[helper.make_tensor_value_info("x", TensorProto.FLOAT,
shape)],
+ outputs=[helper.make_tensor_value_info("y", TensorProto.FLOAT,
shape)],
+ )
+ model = helper.make_model(graph, producer_name="softplus_large_values")
+ inputs = {
+ "x": np.linspace(-10.0, 80.0, np.prod(shape),
dtype="float32").reshape(shape),
+ }
+ check_correctness(model, inputs=inputs, opset=14, rtol=1e-6, atol=1e-6)
+
+
+def test_softplus_float64_accuracy():
+ """float64 Softplus must equal log(exp(x) + 1) across the threshold region.
+ Regression: the frontend used to clamp to the linear function x for x > 20,
+ losing ~1.9e-9 at x = 20.1. onnxruntime's CPU EP has no float64 Softplus,
so
+ the expected value is computed with the numerically stable form
+ max(x, 0) + log1p(exp(-|x|)), which equals log(exp(x) + 1)
mathematically."""
+ xs = np.array([19.0, 20.0, 20.1, 21.0, 25.0, 30.0, 40.0], dtype=np.float64)
+ expected = np.maximum(xs, 0.0) + np.log1p(np.exp(-np.abs(xs)))
+
+ graph = helper.make_graph(
+ [helper.make_node("Softplus", ["x"], ["y"])],
+ "softplus_float64",
+ inputs=[helper.make_tensor_value_info("x", TensorProto.DOUBLE,
[len(xs)])],
+ outputs=[helper.make_tensor_value_info("y", TensorProto.DOUBLE,
[len(xs)])],
+ )
+ model = helper.make_model(graph, producer_name="softplus_float64")
+ model.opset_import[0].version = 14
+
+ tvm_model = from_onnx(model, opset=14, keep_params_in_input=True)
+ tvm_model = relax.transform.DecomposeOpsForInference()(tvm_model)
+ tvm_model = relax.transform.LegalizeOps()(tvm_model)
+ tvm_model, params = relax.frontend.detach_params(tvm_model)
+
+ with tvm.transform.PassContext(opt_level=3):
+ ex = tvm.compile(tvm_model, target="llvm")
+ vm = relax.VirtualMachine(ex, tvm.cpu())
+
+ vm.set_input("main", xs)
+ vm.invoke_stateful("main")
+ tvm_output = vm.get_outputs("main").numpy()
+
+ np.testing.assert_allclose(tvm_output, expected, rtol=1e-12, atol=1e-12)
+
+
def test_thresholded_relu():
model = make_unary_model("ThresholdedRelu", [2, 3])
tvm_model = from_onnx(model, keep_params_in_input=True)
