danielalyoshin opened a new pull request, #44505: URL: https://github.com/apache/superset/pull/44505
<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly --> ### SUMMARY <!--- Describe the change below, including rationale and design decisions --> The `d1` extra is one of the connector extras still held back from SQLAlchemy 2. `UPDATING.md` lists it under "the upstream dialect package has no SQLAlchemy 2.0 support yet at all", and `superset/db_engine_specs/d1.py` carries a `known_incompatibilities` entry whose `tracking_url` points at our repo. This PR lifts that block. I am one of the maintainers of the `sqlalchemy-cf-d1` packages. What changed on our side: * `sqlalchemy-d1` 0.2.0 supports SQLAlchemy 2. It keeps the `d1://` scheme and is a thin layer over the community dialect [`sqlalchemy-cloudflare-d1`](https://github.com/CollierKing/sqlalchemy-cloudflare-d1). It is on PyPI: [sqlalchemy-d1 0.2.0](https://pypi.org/project/sqlalchemy-d1/0.2.0/). * `superset-engine-d1` and `dbapi-d1` are retired. The engine spec of record is the one in this repo. It has taken priority over the external one for `d1://` since 6.1.0. `sqlalchemy-d1` 0.2.0 gets its DBAPI from `sqlalchemy-cloudflare-d1`, so nothing needs `dbapi-d1` any more. What this PR changes: * `pyproject.toml`: the `d1` extra installs `sqlalchemy-d1>=0.2.0, <1` only. * `superset/db_engine_specs/d1.py`: * `pypi_packages` names `sqlalchemy-d1`, and the `known_incompatibilities` entry is removed. * `install_instructions` is `pip install "apache-superset[d1]"` with a `version_requirements` note, as #44285 did for Presto. A bare `pip install sqlalchemy-d1` would call the requirement met on an environment that still has 0.1.0. * `allows_sql_comments = False`. The DBAPI in `sqlalchemy-cloudflare-d1` only reports column names when the statement text starts with `SELECT`, `PRAGMA` or `WITH`, so a query with a leading comment came back as rows without a cursor description and SQL Lab failed on it. With the flag off, Superset strips the comment first. * `UPDATING.md`: `d1` is taken off the list of blocked extras, and a note tells existing installs to uninstall the two retired packages. * `docs/src/data/databases.json`: the matching lines for D1, edited by hand as #42125 and #44285 did. * `tests/unit_tests/db_engine_specs/test_d1.py`: new. Covers the comment stripping, the metadata, and that the type names the dialect reports (`DATETIME`, `TIMESTAMP`, `DATE`, `TIME`, `BOOLEAN`) map to temporal and boolean columns. D1 keeps internal `_cf_*` tables (for example `_cf_KV`) in every database. The `sqlalchemy-d1` dialect hides them, so the spec does not need to. Existing `d1://` connections keep working. The connection string does not change. **Request: please consider this for 7.0.** The `7.0` branch is the first on SQLAlchemy 2 and still lists all three packages in the `d1` extra. `superset-engine-d1` 0.1.0 and `dbapi-d1` 0.1.0 both require Python below 3.12, so without this change `pip install apache-superset[d1]` on 7.0 only works on Python 3.11. The change is small and cherry-picks onto `7.0` without conflicts (checked against `c636115d40`). If a committer agrees, a `v7.0` label would be appreciated. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> **1. Test connection succeeds for a `d1://` database (connection string blurred).** <img width="1440" height="757" alt="01-test-connection" src="https://github.com/user-attachments/assets/ef6161d4-031e-4e0e-873a-6028f613c32b"; /> **2. SQL Lab lists the tables and the view but not `_cf_KV`, and a `DATETIME` column holding a number, an empty string and `09/21/2026` still previews.** <img width="1440" height="757" alt="02-sql-lab-table-list-and-messy-dates" src="https://github.com/user-attachments/assets/99514f67-ff6b-4baa-91f1-154cb679f1b8"; /> **3. A dataset from a D1 table marks its `DATETIME` and `DATE` columns temporal on its own.** <img width="1440" height="806" alt="03-dataset-temporal-columns" src="https://github.com/user-attachments/assets/be3e7d7e-266f-4e58-9bfb-249e913b12b9"; /> **4. A line chart with a daily time grain.** <img width="1440" height="806" alt="04-daily-line-chart" src="https://github.com/user-attachments/assets/e8703d26-8dc9-4542-9ad3-1f2a120ce3d1"; /> ### TESTING INSTRUCTIONS <!--- Required! What steps can be taken to manually verify the changes? --> Unit tests: ```bash pytest tests/unit_tests/db_engine_specs/test_d1.py ``` By hand, against a real D1 database: 1. `pip install "apache-superset[d1]"`, or `pip install "sqlalchemy-d1>=0.2.0"` into an existing Superset environment. Do not install `superset-engine-d1` or `dbapi-d1`. 2. Add a database with `d1://{cloudflare_account_id}:{cloudflare_api_token}@{cloudflare_d1_database_id}`. **Test connection** succeeds. 3. In SQL Lab, `SELECT * FROM <table> WHERE 1=0` returns the column headers and no rows. 4. Create a dataset from a table with `DATETIME` and `DATE` columns. Both are marked temporal. A line chart with a daily time grain renders. 5. `_cf_KV` is not in the table list. 6. In SQL Lab, a query that starts with a `-- comment` line runs and shows its columns. Results. I ran the full list on the final state of this branch on 2026-09-21, in the `docker-compose-light.yml` stack (Python 3.11.14, SQLAlchemy 2.0.52), with `sqlalchemy-d1` 0.2.0 and `sqlalchemy-cloudflare-d1` 0.4.0 installed through `docker/requirements-local.txt`, against a real D1 database. `superset-engine-d1` and `dbapi-d1` were not installed. | Check | Result | |-------|--------| | `get_engine_spec("d1")` returns `CloudflareD1EngineSpec`, no spec load warnings in the log | pass | | Cloudflare D1 is in the database list, **Test connection** succeeds | pass | | SQL Lab: `SELECT 1` | pass | | SQL Lab: a query with zero rows returns its column names and no error | pass | | SQL Lab: a query that starts with a `--` or a `/* */` comment runs and returns its columns | pass | | SQL Lab: a bad query gives a readable error | pass | | The table list does not show `_cf_KV` | pass | | A table whose `DATETIME` column holds a number, an empty string and `09/21/2026` still previews | pass | | A dataset from a table marks its `DATETIME` and `DATE` columns temporal on its own | pass | | A line chart with a daily time grain renders, also with a time range filter | pass | | A view is listed and works as a dataset | pass | | A dashboard with a D1 chart loads and refreshes | pass | ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
