dosubot[bot] commented on issue #37925: URL: https://github.com/apache/superset/issues/37925#issuecomment-3889918178
<!-- Answer --> You're right—what you're seeing can't be explained by a unified offset or time zone config. Superset's chart pipeline applies normalization and offset logic to datetime columns that SQL Lab does not, but those should always result in consistent, uniform shifts if configured correctly. The inconsistent, per-row time shifts you describe point to a bug in the backend's time-handling logic for visualizations. In the master branch, chart queries go through a normalization step (normalize_dttm_col) that applies dataset offsets and time shifts, and—if time comparisons are enabled—joins additional offset DataFrames using generated join keys based on the time grain. If there's a bug in how these join keys are generated or how offsets are parsed/applied, it can cause non-uniform, even non-deterministic, shifts per data point, exactly as you describe. This logic is separate from SQL Lab, which just returns raw database values without any of these transformations. Recent changes to time offset and comparison logic (including PRs like #30002 and #37698) have made this code more complex, especially around parsing human-readable time shifts and joining offset DataFrames. If the join columns or offset calculations are misaligned, you can get the kind of inconsistent results you're seeing ([backend code reference](https://github.com/apache/superset/blob/master/superset/common/query_context_processor.py), [offset normalization](https://github.com/apache/superset/blob/master/superset/utils/core.py), [date parsing](https://github.com/apache/superset/blob/master/superset/utils/date_parser.py)). To debug, I'd suggest: inspect the DataFrame before and after normalization/offset application in the backend, check if offsets are being double-applied or misapplied, and review how join keys are generated for your time grain. If you can, try to isolate a minimal dataset and chart config that reproduces the issue, and see if the problem persists with different time grains or after disabling time comparisons. This looks like a regression or subtle bug in the time offset/join logic for chart queries, not a config issue. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7bf9f2ee-aef5-4dcc-abe7-8f1947f7e323?feedback_type=other)</sup> [](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github) [](https://cursor.com/link/prompt?text=You%27re%20right%E2%80%94what%20you%27re%20seeing%20can%27t%20be%20explained%20by%20a%20unified%20offset%20or%20time%20zone%20config.%20Superset%27s%20chart%20pipeline%20applies%20normalization%20and%20offset%20logic%20to%20datetime%20columns%20that%20SQL%20Lab%20does%20not%2C%20but%20those%20should%20always%20result%20in%20consistent%2C%20uniform%20shifts%20if%20configured%20correctly.%0A%0AThe%20inconsistent%2C%20per-row%20time%20shifts%20you%20describe%20point%20to%20a%20bug%20in%20the%20backend%27s%20time-handling%20logic%20for%20visualizations.%20In%20the%20master%20branch%2C%20chart%20queries%20go%20through%20a%20normalization%20step%20%28normalize_dttm_col%29%20that%20applies%20dataset%20offsets%20and%20time%20shifts%2C%20and%E2%80%94if%20ti me%20comparisons%20are%20enabled%E2%80%94joins%20additional%20offset%20DataFrames%20using%20generated%20join%20keys%20based%20on%20the%20time%20grain.%20If%20there%27s%20a%20bug%20in%20how%20these%20join%20keys%20are%20generated%20or%20how%20offsets%20are%20parsed/applied%2C%20it%20can%20cause%20non-uniform%2C%20even%20non-deterministic%2C%20shifts%20per%20data%20point%2C%20exactly%20as%20you%20describe.%20This%20logic%20is%20separate%20from%20SQL%20Lab%2C%20which%20just%20returns%20raw%20database%20values%20without%20any%20of%20these%20transformations.%0A%0ARecent%20changes%20to%20time%20offset%20and%20comparison%20logic%20%28including%20PRs%20like%20%2330002%20and%20%2337698%29%20have%20made%20this%20code%20more%20complex%2C%20especially%20around%20parsing%20human-readable%20time%20shifts%20and%20joining%20offset%20DataFrames.%20If%20the%20join%20columns%20or%20offset%20calculations%20are%20misaligned%2C%20you%20can%20get%20the%20kind%20of%20inconsistent%20results%20you%27re%20see ing%20%28%5Bbackend%20code%20reference%5D%28https%3A//github.com/apache/superset/blob/master/superset/common/query_context_processor.py%29%2C%20%5Boffset%20normalization%5D%28https%3A//github.com/apache/superset/blob/master/superset/utils/core.py%29%2C%20%5Bdate%20parsing%5D%28https%3A//github.com/apache/superset/blob/master/superset/utils/date_parser.py%29%29.%0A%0ATo%20debug%2C%20I%27d%20suggest%3A%20inspect%20the%20DataFrame%20before%20and%20after%20normalization/offset%20application%20in%20the%20backend%2C%20check%20if%20offsets%20are%20being%20double-applied%20or%20misapplied%2C%20and%20review%20how%20join%20keys%20are%20generated%20for%20your%20time%20grain.%20If%20you%20can%2C%20try%20to%20isolate%20a%20minimal%20dataset%20and%20chart%20config%20that%20reproduces%20the%20issue%2C%20and%20see%20if%20the%20problem%20persists%20with%20different%20time%20grains%20or%20after%20disabling%20time%20comparisons.%0A%0AThis%20looks%20like%20a%20regression%20or%20subtle%20bug%20in%20the% 20time%20offset/join%20logic%20for%20chart%20queries%2C%20not%20a%20config%20issue.) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/issues/37925) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
