mikebridge opened a new pull request, #45126:
URL: https://github.com/apache/superset/pull/45126
### SUMMARY
Semantic-view chart data loses integer precision when an integer metric or
dimension column contains nulls. Arrow's `to_pandas()` turns a nullable `int64`
column into `float64`, so a value such as `9007199254740993` comes back as
`9007199254740992`.
**Plain chart data stays exact.** The semantic mapper converts provider
results with `to_pandas(integer_object_nulls=True)` for the main query, each
time-offset query and the final result conversion. Nullable integer columns
therefore keep Python `int` values, and the existing JSON encoding returns:
- integers inside JavaScript's safe range as numbers;
- integers outside it as exact strings;
- nulls as nulls.
That covers signed 64-bit values up to `2**63 - 1`, including
time-comparison columns.
**Post-processed results use float arithmetic (documented limitation).**
Exact integers make metric columns `object` dtype. SQL datasets convert metrics
to numbers before chart calculations (`df_metrics_to_num`); semantic views now
do the same before post-processing. The conversion uses a shared
`df_columns_to_num` helper and covers the metric columns and their time-offset
variants. Contribution, diff, rolling and cumulative calculations therefore see
numbers and follow normal floating-point rounding, as for SQL datasets. Without
this step, contribution and diff on a nullable integer metric failed with a 400.
**Contribution with a separate totals query.** A totals query that isn't
post-processed keeps a `Decimal` metric as `Decimal`, while the main query's
metric has been converted to `float`. `contribution` now converts a `Decimal`
total to `float` when the column it divides is numeric. Columns that are still
`Decimal` are divided as before.
Not converted: plain output, the totals query's output, and dimension
columns. The behaviour is documented in *Creating your first dashboard →
Integer precision in semantic views*, including that previously cached results
keep rounded values until refreshed.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable (chart-data response values).
| Scenario | Before | After |
| --- | --- | --- |
| Nullable `int64` metric `9007199254740993` | `9007199254740992` (rounded)
| `"9007199254740993"` (exact string) |
| Contribution or diff on a nullable integer metric | worked on rounded
floats | works; float arithmetic, as for SQL datasets |
| Decimal metric, contribution with a separate totals query | worked | still
works (regression covered) |
### TESTING INSTRUCTIONS
```bash
pytest tests/unit_tests/semantic_layers/integer_precision_test.py \
tests/unit_tests/semantic_layers/ \
tests/unit_tests/pandas_postprocessing/
```
The tests go through `POST /api/v1/chart/data` with a mocked provider:
- exact large positive and negative values and `2**63 - 1`, with and without
a time offset;
- contribution and diff on nullable integer metrics;
- contribution with a separate totals query, for decimal, integer and float
metrics.
Local checks: semantic-layer and post-processing tests, 1006 passed. Branch
pre-commit hooks passed, including MyPy. The whole unit suite (`TZ=UTC`) had
22,429 passed, with three local-environment baseline failures in unchanged
numeric-contract and translation-fixture tests.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `SEMANTIC_LAYERS`
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
