dosubot[bot] commented on issue #37925: URL: https://github.com/apache/superset/issues/37925#issuecomment-3948566801
<!-- Answer --> Thanks for the clarification. Based on my investigation, this does appear to be a distinct bug from #37947, though they share the same root cause: **chart queries apply datetime normalization transformations that SQL Lab does not**. **Root Cause**: The inconsistent, non-uniform time shifts (+16.5h, +10h, +14.33h) you're seeing suggest multiple offset layers are being conditionally applied based on data point characteristics. Charts invoke [`normalize_dttm_col()`](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/utils/core.py#L1668-L1711) through the [QueryContextProcessor](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/common/query_context_processor.py#L287), while SQL Lab bypasses this entirely and returns raw database values. **Why non-uniform offsets occur**: The normalization pipeline applies up to three offset layers sequentially: 1. Datasource offset (hours) via `datasource.offset` 2. Query time shift via `parse_human_timedelta()` 3. Time comparison offsets during [join column generation](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/common/query_context_processor.py#L698-L732) If any of these layers are applied conditionally (e.g., based on format detection or aggregation boundaries), you get inconsistent shifts per row. **Relevant recent regressions**: Several PRs have touched this code path recently: - [PR #37979](https://github.com/apache/superset/pull/37979) - Fixed timezone conversion bugs - [PR #37111](https://github.com/apache/superset/pull/37111) - Fixed time comparison join key regression - [PR #35042](https://github.com/apache/superset/pull/35042) - Refactored `normalize_dttm_col` format detection **Triage Recommendation**: This should be labeled as a **bug** with **high priority** since it breaks time-based visualization reliability. It's likely a regression in the datetime normalization logic on master. To help debug further, could you share: 1. Your dataset's offset configuration (if any) 2. Whether time comparisons are enabled on these charts 3. The time grain setting you're using <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/00aca72f-69b0-44df-9b6a-a5c0e2e4a2ba?feedback_type=other)</sup> [](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=Thanks%20for%20the%20clarification.%20Based%20on%20my%20investigation%2C%20this%20does%20appear%20to%20be%20a%20distinct%20bug%20from%20%2337947%2C%20though%20they%20share%20the%20same%20root%20cause%3A%20%2A%2Achart%20queries%20apply%20datetime%20normalization%20transformations%20that%20SQL%20Lab%20does%20not%2A%2A.%0A%0A%2A%2ARoot%20Cause%2A%2A%3A%20The%20inconsistent%2C%20non-uniform%20time%20shifts%20%28%2B16.5h%2C%20%2B10h%2C%20%2B14.33h%29%20you%27re%20seeing%20suggest%20multiple%20offset%20layers%20are%20being%20conditionally%20applied%20based%20on%20data%20point%20characteristics.%20Charts%20invoke%20%5B%60normalize_dttm_col%28%29%60%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/utils/core. py%23L1668-L1711%29%20through%20the%20%5BQueryContextProcessor%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/common/query_context_processor.py%23L287%29%2C%20while%20SQL%20Lab%20bypasses%20this%20entirely%20and%20returns%20raw%20database%20values.%0A%0A%2A%2AWhy%20non-uniform%20offsets%20occur%2A%2A%3A%20The%20normalization%20pipeline%20applies%20up%20to%20three%20offset%20layers%20sequentially%3A%0A1.%20Datasource%20offset%20%28hours%29%20via%20%60datasource.offset%60%0A2.%20Query%20time%20shift%20via%20%60parse_human_timedelta%28%29%60%0A3.%20Time%20comparison%20offsets%20during%20%5Bjoin%20column%20generation%5D%28https%3A//github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/common/query_context_processor.py%23L698-L732%29%0A%0AIf%20any%20of%20these%20layers%20are%20applied%20conditionally%20%28e.g.%2C%20based%20on%20format%20detection%20or%20aggregation%20boundaries%29%2C%20you%20get%20inconsistent% 20shifts%20per%20row.%0A%0A%2A%2ARelevant%20recent%20regressions%2A%2A%3A%20Several%20PRs%20have%20touched%20this%20code%20path%20recently%3A%0A-%20%5BPR%20%2337979%5D%28https%3A//github.com/apache/superset/pull/37979%29%20-%20Fixed%20timezone%20conversion%20bugs%0A-%20%5BPR%20%2337111%5D%28https%3A//github.com/apache/superset/pull/37111%29%20-%20Fixed%20time%20comparison%20join%20key%20regression%0A-%20%5BPR%20%2335042%5D%28https%3A//github.com/apache/superset/pull/35042%29%20-%20Refactored%20%60normalize_dttm_col%60%20format%20detection%0A%0A%2A%2ATriage%20Recommendation%2A%2A%3A%20This%20should%20be%20labeled%20as%20a%20%2A%2Abug%2A%2A%20with%20%2A%2Ahigh%20priority%2A%2A%20since%20it%20breaks%20time-based%20visualization%20reliability.%20It%27s%20likely%20a%20regression%20in%20the%20datetime%20normalization%20logic%20on%20master.%20To%20help%20debug%20further%2C%20could%20you%20share%3A%0A1.%20Your%20dataset%27s%20offset%20configuration%20%28if%20any%29%0A2.%20Whether%20time%20c omparisons%20are%20enabled%20on%20these%20charts%0A3.%20The%20time%20grain%20setting%20you%27re%20using) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/issues/37925) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
