sancho11 opened a new issue, #42788: URL: https://github.com/apache/superset/issues/42788
### Bug description ## Bug description A deck.gl "Multiple Layers" (`deck_multi`) chart renders only the base map, with none of its configured layers, even though every one of those layers renders correctly as a standalone chart. There are two independent root causes behind that single symptom. I'm filing them together because they live in the same code path: the way `deck_multi` loads its sub-layers; and a single PR can address both. Happy to split them if maintainers prefer. The shared context: `deck_multi` is the only deck.gl chart still declared with `useLegacyApi: true` ([`Multi/index.ts`](https://github.com/apache/superset/blob/master/superset-frontend/plugins/preset-chart-deckgl/src/Multi/index.ts)). Every other layer type (`deck_scatter`, `deck_polygon`, `deck_path`, …) goes through `/api/v1/chart/data`. That is exactly why the layers work standalone and fail here: only `deck_multi` still exercises the legacy `explore_json` + `viz.py` path. Both causes below are therefore concrete symptoms of the migration tracked in #41047 ("Make the Deck.GL plugin non-legacy"), whose first item is migrating the plugins from the legacy API to V1. Completing that would remove both by construction. This report is about what breaks in the meantime, since the failure is silent and hard to diagnose. --- ### Cause 1: sub-layer requests are incompatible with `GLOBAL_ASYNC_QUERIES` To draw its sub-layers, `Multi.tsx` fetches each one from `/superset/explore_json/` with a bare `SupersetClient.get`: https://github.com/apache/superset/blob/master/superset-frontend/plugins/preset-chart-deckgl/src/Multi/Multi.tsx#L342-L344 ```ts SupersetClient.get({ endpoint: url }) .then(({ json }) => { const layer = createLayerFromData(subsliceCopy, json); ``` That request bypasses the chart pipeline, so it has no access to the async-query machinery in `src/middleware/asyncEvent.ts`, a Redux middleware, not reachable from a plugin. Meanwhile `explore_json` short-circuits into the async path: https://github.com/apache/superset/blob/master/superset/views/core.py#L361-L392 ```python if ( is_feature_enabled("GLOBAL_ASYNC_QUERIES") and response_type == ChartDataResultFormat.JSON ): # First, look for the chart query results in the cache. with contextlib.suppress(CacheLoadError): viz_obj = get_viz(..., force_cached=True, force=force) payload = viz_obj.get_payload() if payload is not None: return self.send_data_payload_response(viz_obj, payload) # Otherwise, kick off a background job to run the chart query. ... return json_success(json.dumps(job_metadata), status=202) ``` So each sub-layer request returns either **200 with the payload**, when the result was already cached, or **202 with `{"channel_id": ..., "job_id": ..., "status": "pending", "result_url": null}`** and no `data` at all. `Multi.tsx` cannot consume the 202 handoff. It passes the job-metadata body straight to the layer generator, which fails on the missing key, and the `.catch` in `loadSingleLayer` swallows it to the console. The layer simply never appears. This makes the bug look random: a layer renders if and only if its result happens to be warm in the data cache. Typically one or two layers show up on a second visit to the chart and everything else vanishes. On a dashboard, where the layer queries carry dashboard filters and are usually cold, nothing renders at all. Worth noting that routing these queries through the async path buys nothing: building the parent chart's payload already runs every sub-layer query synchronously, since `DeckGLMultiLayer.get_data()` calls `viz_instance.get_payload()` for each slice. The work is already done by the time the browser issues these requests. --- ### Cause 2: a non-string `line_column` raises, and takes the whole chart with it `DeckPathViz.get_properties` assumes the driver returned a JSON string: https://github.com/apache/superset/blob/master/superset/viz.py#L2369 ```python line_type = self.form_data["line_type"] deser = self.deser_map[line_type] line_column = self.form_data["line_column"] path = deser(data[line_column]) ``` With `line_type: "json"`, `deser` is `json.loads`. psycopg2 returns a `jsonb` column already parsed into a `list`/`dict`, and a `bytea` column as a `memoryview`. `simplejson.loads` rejects anything that is not `str` with a single, misleading message: ```python if _PY3 and not isinstance(s, str): raise TypeError("Input string must be text, not bytes") ``` The message says "bytes", but the value is usually a `list`; which sent me looking for an encoding problem that did not exist. `None` (a NULL row) and an unrecognized `line_type` such as `zipcode` (absent from `deser_map`, so `KeyError`) fail here too. The client-side transform for the same layer already handles every one of these shapes, including GeoJSON `Polygon` / `MultiPolygon` / `Feature`: https://github.com/apache/superset/blob/master/superset-frontend/plugins/preset-chart-deckgl/src/layers/Polygon/transformProps.ts ```ts const parsed = typeof rawPolygonData === 'string' ? JSON.parse(rawPolygonData) : rawPolygonData; ``` On top of that, `DeckGLMultiLayer.get_data()` builds every sub-slice payload with no guard: https://github.com/apache/superset/blob/master/superset/viz.py#L1758 ```python viz_instance = viz_class(datasource=slc.datasource, form_data=form_data) payload = viz_instance.get_payload() ``` So the exception propagates out of the parent chart's request. The client gets a 500 and no `slices` at all, and the chart collapses to a bare base map; including the layers that were perfectly fine. One misconfigured or unusually-typed layer takes out everything. --- ### Reproduction steps **For cause 1:** 1. Enable `GLOBAL_ASYNC_QUERIES` (with the Celery worker and cache backend configured). 2. Create two or more deck.gl layer charts (e.g. a Scatter and an Arc) and confirm each renders on its own. 3. Create a deck.gl "Multiple Layers" chart including both. 4. Open it in Explore, or add it to a dashboard and load the dashboard. 5. Open DevTools → Network and filter on `explore_json`. **For cause 2:** 1. On PostgreSQL, create a table with a `jsonb` column holding polygon coordinates, e.g. `[[lon, lat], [lon, lat], ...]`. 2. Create a deck.gl Polygon chart on it, with `line_column` set to that column and `line_type` = `json`. It renders correctly. 3. Create a deck.gl "Multiple Layers" chart including that Polygon layer. 4. Open the multi-layer chart. ### Expected results The multi-layer chart renders all its configured sub-layers, matching what each standalone layer chart shows. ### Actual results Only the base map renders. **Cause 1:** the `explore_json` requests for the missing sub-layers return `202` with job metadata and no `data`. Per missing layer, the browser console shows: ``` Error loading layer for slice <id>: TypeError: can't access property "features", data is undefined ``` Superset's logs show the cache probe failing for each sub-layer that did not render: ``` WARNING:superset.viz:force_cached (viz.py): value not found for cache key cc248d487cd0963e9ca42faad689b33841110781d6f7dc1534a62915fa99cf25 WARNING:superset.viz:force_cached (viz.py): value not found for cache key 7617950abf012ed1478ec3580e2d9ca37d6b3da8b97cc13b4bb6d38e1c8971ce ``` and, for the one that did: ``` INFO:superset.viz:Serving from cache ``` **Cause 2:** HTTP 500 and `Data error: Input string must be text, not bytes`, with the stacktrace below. ### Screenshots/recordings ## Cause 1: sub-layer requests are incompatible with GLOBAL_ASYNC_QUERIES <img width="979" height="519" alt="Image" src="https://github.com/user-attachments/assets/caf60b82-de49-452d-b995-c4fdac883026"; /> At this second image we should see both the arc layer and a heatmap layer but it did not render. <img width="964" height="665" alt="Image" src="https://github.com/user-attachments/assets/2fc63a5b-96cf-4635-8e9d-923960eefd18"; /> ## Cause 2: a non-string line_column raises, and takes the whole chart with it <img width="1162" height="519" alt="Image" src="https://github.com/user-attachments/assets/28360bc5-787d-4369-8a9a-a95f08a805b8"; /> ### Superset version master / latest-dev ### Python version 3.11 ### Node version 18 or greater ### Browser Firefox ### Additional context **Feature flags:** `GLOBAL_ASYNC_QUERIES` enabled. Disabling it makes cause 1 disappear entirely, which confirms that diagnosis. **Environment:** Docker, gunicorn sync workers, Redis cache + Celery, PostgreSQL/TimescaleDB analytics database via psycopg2. Stacktrace for cause 2: ``` ERROR:superset.views.error_handling:Input string must be text, not bytes Traceback (most recent call last): File "/app/superset/views/error_handling.py", line 125, in wraps return f(self, *args, **kwargs) File "/app/superset/views/core.py", line 374, in explore_json payload = viz_obj.get_payload() File "/app/superset/viz.py", line 1793, in get_payload payload = super().get_payload(query_obj) File "/app/superset/viz.py", line 522, in get_payload payload["data"] = self.get_data(df) File "/app/superset/viz.py", line 1758, in get_data payload = viz_instance.get_payload() File "/app/superset/viz.py", line 522, in get_payload payload["data"] = self.get_data(df) File "/app/superset/viz.py", line 2388, in get_data return super().get_data(df) File "/app/superset/viz.py", line 2103, in get_data feature = self.get_properties(data) File "/app/superset/viz.py", line 2446, in get_properties super().get_properties(data) File "/app/superset/viz.py", line 2376, in get_properties path = deser(data[line_column]) File "/app/superset/utils/json.py", line 256, in loads return simplejson.loads( File "/app/.venv/lib/python3.11/site-packages/simplejson/decoder.py", line 418, in raw_decode raise TypeError("Input string must be text, not bytes") TypeError: Input string must be text, not bytes ``` The two `viz.py:522` frames show the nesting: the outer one is the `deck_multi` payload, the inner one the polygon sub-slice whose failure propagates all the way out. ### Suggested fixes **For cause 1**, roughly in order of how well each addresses the underlying design: 1. **Serve the sub-layer data from the parent payload.** `DeckGLMultiLayer.get_data()` already builds every sub-slice payload; it merges the features by `viz_type` rather than keying them by `slice_id`. Emitting them per slice would let `Multi.tsx` drop the N extra requests entirely, and the parent chart request already handles `GLOBAL_ASYNC_QUERIES` correctly because it goes through the normal chart pipeline. The catch is that `Multi.tsx` applies its own `layerFilterScope` logic to `adhoc_filters` in Explore mode, which `_apply_layer_filtering` on the backend does not replicate identically, so the filter semantics would need reconciling first. 2. **Have `Multi.tsx` mark its sub-layer requests and let `explore_json` answer them synchronously.** Small and low-risk, and it adds no query load for the reason described above; the parent request has already populated the cache these requests read. This is what I'm running locally. 3. **Teach `Multi.tsx` the async protocol.** This would mean duplicating the `asyncEvent` middleware inside a plugin, including sharing the global event stream, which seems clearly wrong. **For cause 2:** 4. Normalize the `line_column` value in `DeckPathViz.get_properties` the way the client-side transform already does; accept `str`, `bytes`/`bytearray`/`memoryview`, an already-parsed `list`, and GeoJSON geometry/Feature mappings, and return an empty path for NULL instead of raising. Falling back to an empty path for an unrecognized `line_type` would also remove the `KeyError` on `zipcode`. 5. Wrap the per-slice `get_payload()` in `DeckGLMultiLayer.get_data()` so a sub-slice that cannot be rendered is logged and skipped, and the remaining layers still render. **Independently of both**, `Multi.tsx` should not treat a payload without `data` as a valid layer payload. `BaseViz.get_payload` only sets `payload["data"]` when the query succeeded, so a failed sub-slice also answers 200 with the reason in `payload["errors"]` and no `data` key. Checking for that and reporting `errors[0].message` turns an opaque `TypeError` into the actual cause; during this investigation it was exactly what masked cause 2 behind a misleading frontend error. I have all of the above running locally against the reproductions described here, and I'm happy to open a PR for whichever direction maintainers prefer. ### Related issues I searched the tracker and did not find either cause reported. Adjacent but distinct: - #41047 (open): "Make the Deck.GL plugin non-legacy". The umbrella migration; both causes here stem from `deck_multi` still being legacy, so finishing it would resolve them. If that work is imminent, this report may be most useful as a set of test cases. - #32642 (open): `deck_multi` does not refresh after the underlying dataset changes. Different failure mode (stale data rather than no data). - #36779 (closed): deck.gl Polygon hover error in 6.0.0. Client-side tooltip path, unrelated to the server-side `line_column` deserialization described here. ### Checklist - [x] I have searched Superset docs and Slack and didn't find a solution to my problem. - [x] I have searched the GitHub issue tracker and didn't find a similar bug report. - [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
