This is an automated email from the ASF dual-hosted git repository.

nsivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git


The following commit(s) were added to refs/heads/master by this push:
     new f5fe1dce2720 feat(lakehouse): add Superset and Zeppelin Spark Connect 
integration (#20467)
f5fe1dce2720 is described below

commit f5fe1dce27206447d7bab2dc60d7735ac3cba80b
Author: Shuo Cheng <[email protected]>
AuthorDate: Fri Nov 6 12:36:11 2026 +0800

    feat(lakehouse): add Superset and Zeppelin Spark Connect integration 
(#20467)
    
    Add an experimental SQLAlchemy dialect and Superset engine spec under
      hudi-lakehouse/superset-spark-connect, backed by the Foundry Spark ADBC
      driver, so BI tools can query and write Hudi tables via Spark Connect
      instead of the Thrift server.
    
      - Dialect supports reflection, temporal filters and offset pagination;
        it is pinned to the spark ADBC driver (driver/URI overrides rejected)
        and rejects parameterized execution.
      - Dockerfile extending Superset 6.1.0 with the adapter and native driver;
        package accepts PyArrow >=16.1,<19.
      - Document Zeppelin 0.12 access via the %python interpreter, plus a
        local Superset/Zeppelin quickstart.
      - Offline dialect tests in CI (PyArrow 16.1 and 18.1); opt-in live tests
        for the adapter, Superset SQL Lab and Zeppelin.
    
      Closes #20466
---
 .github/workflows/agent_gateway_ci.yml             |  18 ++
 hudi-lakehouse/README.md                           |   7 +-
 hudi-lakehouse/charts/hudi-spark-connect/README.md |  70 ++++++--
 hudi-lakehouse/superset-spark-connect/Dockerfile   |  24 +++
 .../superset-spark-connect/QUICKSTART.md           | 163 +++++++++++++++++
 hudi-lakehouse/superset-spark-connect/README.md    | 146 +++++++++++++++
 .../hudi_spark_connect/__init__.py                 |  15 ++
 .../hudi_spark_connect/dialect.py                  | 180 +++++++++++++++++++
 .../hudi_spark_connect/engine_spec.py              |  72 ++++++++
 .../superset-spark-connect/pyproject.toml          |  33 ++++
 .../superset-spark-connect/test_dialect.py         | 137 ++++++++++++++
 .../superset-spark-connect/test_engine_spec.py     |  83 +++++++++
 hudi-lakehouse/superset-spark-connect/test_live.py | 138 +++++++++++++++
 .../superset-spark-connect/test_superset_live.py   | 196 +++++++++++++++++++++
 .../superset-spark-connect/test_zeppelin_live.py   |  93 ++++++++++
 15 files changed, 1362 insertions(+), 13 deletions(-)

diff --git a/.github/workflows/agent_gateway_ci.yml 
b/.github/workflows/agent_gateway_ci.yml
index 6e16575e3514..d2074bb76f37 100644
--- a/.github/workflows/agent_gateway_ci.yml
+++ b/.github/workflows/agent_gateway_ci.yml
@@ -43,6 +43,24 @@ permissions:
   contents: read
 
 jobs:
+  superset-spark-connect-tests:
+    name: Superset Spark Connect dialect tests (PyArrow ${{ matrix.pyarrow }})
+    runs-on: ubuntu-24.04
+    strategy:
+      matrix:
+        pyarrow: ['16.1.0', '18.1.0']
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+          cache: pip
+          cache-dependency-path: 
hudi-lakehouse/superset-spark-connect/pyproject.toml
+      - name: Install adapter
+        run: pip install ./hudi-lakehouse/superset-spark-connect 'pyarrow==${{ 
matrix.pyarrow }}'
+      - name: Run offline dialect tests
+        run: python -m unittest discover -s 
hudi-lakehouse/superset-spark-connect -p 'test_dialect.py' -v
+
   gateway-tests:
     name: gateway pytest + lint (python ${{ matrix.python }})
     runs-on: ubuntu-24.04
diff --git a/hudi-lakehouse/README.md b/hudi-lakehouse/README.md
index d362eec1f2ca..974bb0751f0a 100644
--- a/hudi-lakehouse/README.md
+++ b/hudi-lakehouse/README.md
@@ -179,8 +179,11 @@ The gateway selects the official PySpark Connect client 
with
 
 See the [BI client 
notes](charts/hudi-spark-connect/README.md#spark-connect-client-tutorials)
 for Superset and Zeppelin connection requirements and ADBC limitations.
-Gateway integration and BI adapters/examples for Superset and other tools will 
follow
-in subsequent PRs, tracked in 
[#20232](https://github.com/apache/hudi/issues/20232).
+The [experimental Superset adapter](superset-spark-connect/README.md) uses 
ADBC over
+Spark Connect. It includes opt-in SQL Lab and chart-data API checks; browser
+dashboard refresh is documented as a separate manual acceptance step.
+Agent and MCP gateway integration is tracked separately in
+[#20232](https://github.com/apache/hudi/issues/20232).
 
 ## Version compatibility notes
 
diff --git a/hudi-lakehouse/charts/hudi-spark-connect/README.md 
b/hudi-lakehouse/charts/hudi-spark-connect/README.md
index fd7c7d45c207..ebbadb1d7eb0 100644
--- a/hudi-lakehouse/charts/hudi-spark-connect/README.md
+++ b/hudi-lakehouse/charts/hudi-spark-connect/README.md
@@ -314,17 +314,65 @@ do not require an LLM. Override the provider as described 
in the gateway README.
 
 ## BI clients
 
-BI adapters/examples for Superset and other tools are tracked
-in [#20232](https://github.com/apache/hudi/issues/20232) and will be delivered 
in follow-up PRs.
-The initial SQLAlchemy adapter is planned under `hudi-lakehouse`; upstreaming 
can
-be evaluated separately after the integration stabilizes.
-
-- **Superset 5.0:** requires a custom SQLAlchemy dialect and engine spec for 
ADBC;
-  neither is included here. The built-in PyHive connector uses Thrift, not 
Connect.
-  See [Superset engine 
specs](https://github.com/apache/superset/blob/5.0.0/superset/db_engine_specs/README.md).
-- **Zeppelin 0.12:** use ADBC from the `%python` interpreter, with the client
-  packages above installed in its Python environment. See the
-  [Python interpreter 
guide](https://zeppelin.apache.org/docs/0.12.0/interpreter/python.html).
+For a local, step-by-step Superset and Zeppelin example, see the
+[BI quickstart](../../superset-spark-connect/QUICKSTART.md).
+
+The [experimental Superset adapter](../../superset-spark-connect/README.md)
+provides a SQLAlchemy dialect and Superset engine spec for Foundry ADBC. Follow
+its installation instructions and application-level checks. Those checks are
+separate from the endpoint acceptance tests above.
+Upstreaming the adapter can be evaluated separately after the integration 
stabilizes.
+
+For Zeppelin 0.12, install the ADBC packages and Foundry driver into the Python
+environment used by the `%python` interpreter. Its official 0.12.0 image uses
+Python 3.9, so the manager 1.12.0 requirements above do not apply to that 
image.
+The following tested versions are installed as that interpreter's user (or at
+system level where they are readable by that user):
+
+```bash
+python3 -m pip install adbc-driver-manager==1.8.0 pyarrow==18.1.0 dbc==0.3.1
+dbc install 'spark=0.2.1'
+```
+
+In a notebook, connect to the endpoint and render a query (replace the table 
with
+an isolated test table):
+
+```python
+%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+with dbapi.connect(driver="spark", db_kwargs={"uri": 
"spark://localhost:15002?api=connect&auth_type=none&tls=false"}, 
autocommit=True) as conn:
+    with conn.cursor() as cur:
+        cur.execute("SELECT region, SUM(amount) AS total FROM 
connect_smoke_example.orders GROUP BY region ORDER BY region")
+        rows = cur.fetchall()
+        z.show(pd.DataFrame(rows, columns=["region", "total"]))
+```
+
+The hostname is relative to the Zeppelin interpreter process. Check the 
rendered
+rows against the test fixture, then run INSERT, UPDATE and DELETE and SELECT 
after
+each mutation in a disposable table. This uses `%python`, not native `%jdbc`.
+The exact rendered output depends on the installed interpreter. A local run of
+Zeppelin 0.12.0's x86_64 image (emulated on ARM), Python 3.9.15, ADBC manager
+1.8.0, PyArrow 18.1.0 and Foundry Spark ADBC 0.2.1 executed this path against a
+local Spark 4.1.3/Hudi 1.2.0 endpoint. It checked INSERT, UPDATE, DELETE and
+readback and rendered the final aggregation.
+
+For a repeatable API-driven check, install `requests` in the test environment,
+set `HUDI_ZEPPELIN_URL` and `HUDI_ZEPPELIN_CONNECT_URI` (the latter must be
+reachable from the Zeppelin interpreter), and run the CRUD and MERGE test:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p 
'test_zeppelin_live.py' -v
+```
+
+Use only a disposable endpoint and notebook server. The Connect URI is stored 
in
+the notebook source, so the test requires `auth_type=none` and rejects 
credentials
+and secret-bearing options. The test creates a unique notebook and table and
+removes them; if execution is interrupted or the outcome
+is ambiguous, wait for the notebook to stop before manually cleaning up the
+printed table name. API-created paragraphs must include a table-results config;
+without it this Zeppelin version returned a table result via API but its 
browser
+UI failed to render the paragraph.
 
 [Foundry Spark ADBC 0.2.1](https://adbc-drivers.org/drivers/spark/) uses 
autocommit.
 
diff --git a/hudi-lakehouse/superset-spark-connect/Dockerfile 
b/hudi-lakehouse/superset-spark-connect/Dockerfile
new file mode 100644
index 000000000000..6f3d03e5d3a0
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/Dockerfile
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+FROM apache/superset:6.1.0
+USER root
+COPY . /tmp/hudi-spark-connect-superset
+RUN uv pip install --python /app/.venv/bin/python 
/tmp/hudi-spark-connect-superset 'dbc==0.3.1' 'pyarrow==18.1.0' \
+    && /app/.venv/bin/dbc install --level system 'spark=0.2.1' \
+    && mkdir -p /app/superset_home/.config \
+    && chown -R superset:superset /app/superset_home/.config \
+    && rm -rf /tmp/hudi-spark-connect-superset
+USER superset
diff --git a/hudi-lakehouse/superset-spark-connect/QUICKSTART.md 
b/hudi-lakehouse/superset-spark-connect/QUICKSTART.md
new file mode 100644
index 000000000000..5ae9d4073393
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/QUICKSTART.md
@@ -0,0 +1,163 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+associated with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Try Superset and Zeppelin with Spark Connect
+
+This local example uses a disposable Hudi table. It targets Docker Desktop on
+macOS, Superset 6.1.0 and Zeppelin 0.12.0. Run shell commands from the root of
+an Apache Hudi checkout containing this adapter. You need Docker Desktop,
+minikube, kubectl, Helm and network access for images and dependencies. 
Allocate
+at least 4 CPUs and 10 GiB of memory to minikube, plus resources for the BI
+containers. First startup can take several minutes. These settings are for a
+trusted local environment, not production.
+
+## 1. Start Spark Connect
+
+Start Docker Desktop, then run:
+
+```bash
+./hudi-lakehouse/scripts/quickstart.sh --spark-connect
+kubectl -n hudi-lakehouse rollout status deployment/hudi-spark-connect 
--timeout=300s
+```
+
+In a separate terminal, leave the port-forward running:
+
+```bash
+kubectl -n hudi-lakehouse port-forward svc/hudi-spark-connect 15002:15002
+```
+
+If minikube reports a missing `minikube` Docker container while the script says
+it is running, the profile may be stale. Check `minikube status` and Docker
+Desktop's current context; repair or recreate the profile only after checking
+whether its existing data is needed. See the [endpoint 
guide](../charts/hudi-spark-connect/README.md)
+for other deployment options.
+
+## 2. Start Superset and connect
+
+Build and initialize a disposable local Superset instance:
+
+```bash
+docker build -t hudi-superset-connect:local 
hudi-lakehouse/superset-spark-connect
+docker run -d --name hudi-bi-superset -p 127.0.0.1:8088:8088 \
+  -e SUPERSET_SECRET_KEY=local-demo-secret-change-for-real-use \
+  -e SUPERSET_HOME=/tmp/superset-home \
+  --entrypoint /bin/sh hudi-superset-connect:local \
+  -c 'mkdir -p "$SUPERSET_HOME" &&
+      superset db upgrade &&
+      superset fab create-admin --username admin --firstname Local --lastname 
Demo \
+        --email [email protected] --password admin &&
+      superset init &&
+      superset run -h 0.0.0.0 -p 8088'
+docker logs -f hudi-bi-superset
+```
+
+Once startup completes, press Ctrl-C to stop following the logs; this does not
+stop the container. Open <http://localhost:8088> and sign in with `admin` / 
`admin`.
+Under **Settings → Database Connections**, click **+ Database**, and select
+**Apache Spark Connect (ADBC)**. On the **Basic** tab, enter a display name and
+the SQLAlchemy URI below. Before clicking **Connect**, open **Advanced → SQL 
Lab**
+and enable **Allow DDL and DML** so the example can create and write its table.
+Use a disposable connection.
+
+```text
+sparkconnect://host.docker.internal:15002/default?api=connect&auth_type=none&tls=false
+```
+
+`host.docker.internal` routes from a Docker Desktop container to the Mac; it is
+not the Kubernetes service name. Test the connection and open **SQL Lab**.
+Run the following statements one at a time, selecting this connection and the
+`default` schema. The table name is reserved for this walkthrough: use a unique
+name if it already exists.
+
+```sql
+CREATE TABLE default.bi_connect_demo (id INT, region STRING, amount INT, ts 
BIGINT)
+USING hudi TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
+```
+
+```sql
+INSERT INTO default.bi_connect_demo VALUES (1, 'East', 100, 1), (2, 'West', 
200, 1);
+```
+
+```sql
+MERGE INTO default.bi_connect_demo t
+USING (SELECT 1 AS id, 'East' AS region, 175 AS amount, CAST(3 AS BIGINT) AS ts
+       UNION ALL SELECT 3, 'North', 300, CAST(3 AS BIGINT)) s
+ON t.id = s.id
+WHEN MATCHED THEN UPDATE SET *
+WHEN NOT MATCHED THEN INSERT *;
+```
+
+```sql
+SELECT id, region, amount, ts FROM default.bi_connect_demo ORDER BY id;
+```
+
+Expected rows: `(1, East, 175, 3)`, `(2, West, 200, 1)` and
+`(3, North, 300, 3)`. Create a dataset from `bi_connect_demo` and a chart 
grouped
+by `region` to explore the data. If a write times out or disconnects, check the
+server and table before retrying: the write may still commit.
+
+## 3. Try Zeppelin
+
+Start Zeppelin and install the client in its Python environment:
+
+```bash
+docker run -d --name hudi-bi-zeppelin -p 127.0.0.1:8080:8080 
apache/zeppelin:0.12.0
+docker exec --user root hudi-bi-zeppelin bash -c \
+  "python3 -m pip install adbc-driver-manager==1.8.0 pyarrow==18.1.0 
dbc==0.3.1 && dbc install --level system 'spark=0.2.1' && chown -R 1000:0 
/opt/zeppelin/.config"
+```
+
+The final `chown` makes the configuration directory created by `dbc` accessible
+to Zeppelin's user in the official image. If the driver registry download 
fails,
+check network access and rerun the `docker exec` command before opening a 
notebook.
+
+Open <http://localhost:8080>, create a notebook and run this `%python` 
paragraph.
+It reads the same table created in Superset and renders the three rows:
+
+```python
+%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+
+uri = 'spark://host.docker.internal:15002?api=connect&auth_type=none&tls=false'
+with dbapi.connect(driver='spark', db_kwargs={'uri': uri}, autocommit=True) as 
conn:
+    with conn.cursor() as cur:
+        cur.execute('SELECT id, region, amount, ts FROM 
default.bi_connect_demo ORDER BY id')
+        z.show(pd.DataFrame(cur.fetchall(), columns=['id', 'region', 'amount', 
'ts']))
+```
+
+The adapter tests also exercise Zeppelin writes and MERGE; see the
+[client guide](README.md) and [Zeppelin test 
instructions](../charts/hudi-spark-connect/README.md#bi-clients).
+Zeppelin uses ADBC from `%python`; this example does not use `%jdbc`.
+
+## 4. Clean up
+
+In Superset SQL Lab, after all queries have completed, run:
+
+```sql
+DROP TABLE default.bi_connect_demo PURGE;
+```
+
+Remove only the two containers created above, and stop the port-forward with
+Ctrl-C in its terminal:
+
+```bash
+docker rm -f hudi-bi-superset hudi-bi-zeppelin
+```
+
+The minikube deployment remains available. See the [local-dev teardown 
guide](../local-dev/README.md)
+before removing the cluster or its data.
diff --git a/hudi-lakehouse/superset-spark-connect/README.md 
b/hudi-lakehouse/superset-spark-connect/README.md
new file mode 100644
index 000000000000..c2023dcfa6dc
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/README.md
@@ -0,0 +1,146 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Spark Connect adapter for Superset
+
+This experimental package provides a SQLAlchemy dialect and a Superset database
+engine spec for the Foundry Spark ADBC driver. It uses Spark Connect, not the
+Spark Thrift Server. It is independent of Hudi-specific SQL and can query other
+Spark tables. The repository's Connect deployment instructions are in
+[the chart README](../charts/hudi-spark-connect/README.md).
+
+For a step-by-step local walkthrough, including Superset and Zeppelin, see the
+[BI quickstart](QUICKSTART.md).
+
+Install the package and native driver in **every** Superset web and worker
+runtime. The driver must be accessible to the user running Superset, not just
+installed in the build user's home. The initial target is Superset 6.1.0,
+Python 3.11, SQLAlchemy 1.4.54, ADBC manager 1.12.0, PyArrow 18.1.0 and
+Foundry Spark ADBC 0.2.1 (the official Superset 6.1.0 container used
+Python 3.10 in the SQL Lab API check):
+
+```bash
+pip install ./hudi-lakehouse/superset-spark-connect 'dbc==0.3.1'
+dbc install 'spark=0.2.1'
+```
+
+The supplied Dockerfile extends the official Superset 6.1.0 image, installs the
+adapter into its existing Python environment and installs the native driver at
+system level for the Superset user:
+
+```bash
+docker build -t hudi-superset-connect:local 
hudi-lakehouse/superset-spark-connect
+```
+
+Use that image for Superset web and worker processes. Configure a Superset 
secret
+and metadata database using the normal Superset deployment instructions. The 
image
+is a local integration example, not a hardened production deployment. It 
replaces
+the base image's PyArrow 16.1.0 with 18.1.0; compatibility with unrelated 
Superset
+features and connectors has not been established. The adapter package accepts
+`pyarrow>=16.1,<19`, so installing it into an existing Superset environment 
does
+not require upgrading PyArrow 16.1.0. The Dockerfile pins 18.1.0 explicitly for
+the reproducible example.
+PyArrow 16.1.0 was also checked with ADBC manager 1.12.0 and Foundry Spark ADBC
+0.2.1: dialect and engine spec tests, Spark/Hudi live tests, and Superset SQL 
Lab
+CRUD/MERGE and chart-data tests passed. This does not establish compatibility
+with every PyArrow version in the range or unrelated Superset connectors.
+
+Restart Superset after installation. Configure the database with a URI such as:
+
+```text
+sparkconnect://localhost:15002/default?api=connect&auth_type=none&tls=false
+```
+
+`localhost` is relative to the Superset process. For an in-cluster client use
+`hudi-spark-connect.hudi-lakehouse.svc:15002`; for a host client, use the
+port-forward in the chart README. This URI is only for a trusted local network.
+The adapter currently rejects credentials in the URL; authenticated endpoints 
and
+catalog selection have not been validated. Its URI path selects the Spark 
schema.
+The adapter always loads the `spark` driver installed by the Superset operator.
+The URI query parameter `driver` is rejected, including `driver=spark`; 
database
+connections cannot select another native library. Install drivers only in 
trusted,
+operator-controlled locations. Do not put credentials in query
+parameters, which can appear in logs.
+
+Use `superset test-db -c '{}'` with the URI to check dialect compatibility. In 
SQL Lab,
+select the schema and table, run a grouped query, then create a dataset and 
chart.
+To test direct writes, enable **Allow DML** on a disposable database 
connection,
+execute INSERT, UPDATE and DELETE against a uniquely named test table, and run 
a
+SELECT after each mutation. Refresh the chart after the writes. Disable Allow 
DML
+when finished. The agent gateway remains read-only; these direct writes bypass 
it.
+
+Parameterized execution and executemany are deliberately rejected: the tested
+Foundry 0.2.1 combination did not support parameter binding. Do not interpolate
+untrusted values into SQL. Driver errors or disconnects can occur after a write
+commits; inspect the table and timeline before retrying. Unsupported complex 
Spark
+types are reported as unknown SQLAlchemy types. Query cancellation, large 
results,
+authentication, concurrency and dashboard behaviour require separate 
validation.
+
+Offline contract tests and an opt-in live test:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p 
'test_dialect.py' -v
+HUDI_SUPERSET_CONNECT_TEST_URI='sparkconnect://localhost:15002/default?tls=false'
 \
+  python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p 
'test_live.py' -v
+```
+
+CI runs only the offline dialect tests, with PyArrow 16.1.0 and 18.1.0.
+The Superset engine spec tests are manual
+and require Superset 6.1.0. After building the example
+image above, run them against the checkout with:
+
+```bash
+docker run --rm -e SUPERSET_SECRET_KEY=local-test-only -e PYTHONPATH=/work \
+  -v "$PWD/hudi-lakehouse/superset-spark-connect:/work:ro" \
+  --entrypoint /app/.venv/bin/python hudi-superset-connect:local \
+  -m unittest discover -s /work -p 'test_engine_spec.py' -v
+```
+
+The live test creates a uniquely named database and Hudi table, checks 
discovery,
+queries and CRUD, and drops only its own table and database. It needs a 
disposable
+server with Hudi extensions and write permissions. A failed cleanup may leave 
the
+named resources behind. Offline tests do not prove Superset UI or Spark 
behaviour.
+
+To exercise Superset's SQL Lab API, install `requests` in the test environment,
+configure a disposable Superset database connection with Allow DML enabled, and
+set `HUDI_SUPERSET_URL`, `HUDI_SUPERSET_USER`, `HUDI_SUPERSET_PASSWORD` and
+`HUDI_SUPERSET_DATABASE_ID`. Optionally set `HUDI_SUPERSET_SCHEMA` (default
+`default`). Then run:
+
+```bash
+python -m unittest discover -s hudi-lakehouse/superset-spark-connect -p 
'test_superset_live.py' -v
+```
+
+The SQL Lab test creates a unique table and Superset dataset, checks MERGE 
updates
+and inserts with readback, verifies chart data
+before and after a forced refresh following an update, then removes both 
resources;
+it reports their names if cleanup fails. Use a disposable environment and
+verify database permissions before enabling DML. The test covers the 
application
+API; it does not automate browser rendering or dashboard refresh.
+
+A local functional run used Superset 6.1.0, Spark Connect 4.1.3, Hudi 1.2.0
+and Foundry ADBC 0.2.1 against both a local container and the Helm-deployed
+endpoint. The SQL Lab API test passed queries, filtering and
+INSERT/UPDATE/DELETE with readback; the chart data API returned updated 
aggregation
+results after a forced refresh. The SQL Lab interface displayed a simple
+query result. In a separate local-container run, a Superset dashboard table 
showed
+East 100 and West 200; after an independent ADBC writer changed East to 150,
+clicking **Refresh dashboard** displayed East 150 and West 200.
+
+These checks used a single local-mode Spark driver and do not establish 
distributed
+execution or production readiness.
diff --git 
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py 
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py
new file mode 100644
index 000000000000..c5a5d9fae9de
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/__init__.py
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git 
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py 
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
new file mode 100644
index 000000000000..74fd2ad88d8f
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
@@ -0,0 +1,180 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""SQLAlchemy dialect for Spark Connect using the Foundry ADBC driver."""
+
+import re
+from urllib.parse import unquote, urlencode
+
+from sqlalchemy import types
+from sqlalchemy.engine import default
+from sqlalchemy.sql.compiler import SQLCompiler
+from sqlalchemy.sql.compiler import IdentifierPreparer
+
+
+class SparkIdentifierPreparer(IdentifierPreparer):
+    def __init__(self, dialect):
+        super().__init__(dialect, initial_quote="`", escape_quote="`")
+
+
+class SparkConnectCompiler(SQLCompiler):
+    def limit_clause(self, select, **kw):
+        text = ""
+        if select._limit_clause is not None:
+            text += "\n LIMIT " + self.process(select._limit_clause, **kw)
+        if select._offset_clause is not None:
+            text += "\n OFFSET " + self.process(select._offset_clause, **kw)
+        return text
+
+
+def spark_type(name):
+    name = name.strip().lower()
+    if name.startswith("decimal("):
+        match = re.fullmatch(r"decimal\((\d+),\s*(\d+)\)", name)
+        if match:
+            return types.Numeric(int(match.group(1)), int(match.group(2)))
+    base = name.split("(", 1)[0]
+    return {
+        "boolean": types.Boolean, "tinyint": types.SmallInteger,
+        "smallint": types.SmallInteger, "int": types.Integer,
+        "integer": types.Integer, "bigint": types.BigInteger,
+        "float": types.Float, "double": types.Float,
+        "string": types.String, "varchar": types.String, "char": types.String,
+        "date": types.Date, "timestamp": types.DateTime,
+        "timestamp_ntz": types.DateTime, "binary": types.LargeBinary,
+    }.get(base, types.NullType)()
+
+
+class SparkConnectDialect(default.DefaultDialect):
+    name = "sparkconnect"
+    driver = "adbc"
+    preparer = SparkIdentifierPreparer
+    statement_compiler = SparkConnectCompiler
+    supports_statement_cache = False
+    supports_native_boolean = True
+    default_paramstyle = "qmark"
+
+    @classmethod
+    def dbapi(cls):
+        from adbc_driver_manager import dbapi
+        return dbapi
+
+    def create_connect_args(self, url):
+        if not url.host:
+            raise ValueError("Spark Connect host is required")
+        if url.username or url.password:
+            raise ValueError("Credentials in SQLAlchemy URL are not supported")
+        options = dict(url.query)
+        if "driver" in options:
+            raise ValueError("The driver URI parameter is not supported; 
install the 'spark' driver on the server")
+        options.setdefault("api", "connect")
+        options.setdefault("auth_type", "none")
+        if options["api"] != "connect":
+            raise ValueError("Only api=connect is supported")
+        host = f"[{url.host}]" if ":" in url.host else url.host
+        authority = f"{host}:{url.port or 15002}"
+        uri = f"spark://{authority}?{urlencode(options)}"
+        return [], {"driver": "spark", "db_kwargs": {"uri": uri}, 
"autocommit": True,
+                    "_schema": unquote(url.database) if url.database else None}
+
+    def connect(self, *args, **kwargs):
+        if kwargs.pop("driver", "spark") != "spark":
+            raise ValueError("Only the 'spark' ADBC driver is supported")
+        schema = kwargs.pop("_schema", None)
+        connection = self.dbapi.connect(*args, driver="spark", **kwargs)
+        if schema:
+            try:
+                with connection.cursor() as cursor:
+                    cursor.execute("USE " + 
self.identifier_preparer.quote_identifier(schema))
+            except BaseException:
+                connection.close()
+                raise
+        return connection
+
+    def do_execute(self, cursor, statement, parameters, context=None):
+        if parameters:
+            raise NotImplementedError("Spark Connect ADBC parameter binding is 
not supported")
+        cursor.execute(statement)
+
+    def do_executemany(self, cursor, statement, parameters, context=None):
+        raise NotImplementedError("Spark Connect ADBC parameter binding is not 
supported")
+
+    def do_rollback(self, dbapi_connection):
+        # Spark does not support SQL transactions; connections use autocommit.
+        pass
+
+    def _get_default_schema_name(self, connection):
+        return connection.exec_driver_sql("SELECT current_database()").scalar()
+
+    def get_schema_names(self, connection, **kw):
+        return [row[0] for row in connection.exec_driver_sql("SHOW DATABASES")]
+
+    def _qualified(self, table, schema=None):
+        quote = self.identifier_preparer.quote_identifier
+        return ".".join(quote(part) for part in (schema, table) if part is not 
None)
+
+    def get_table_names(self, connection, schema=None, **kw):
+        sql = "SHOW TABLES"
+        if schema:
+            sql += " IN " + self.identifier_preparer.quote_identifier(schema)
+        views = set(self.get_view_names(connection, schema=schema))
+        return [row[1] for row in connection.exec_driver_sql(sql) if not 
row[2] and row[1] not in views]
+
+    def get_view_names(self, connection, schema=None, **kw):
+        sql = "SHOW VIEWS"
+        if schema:
+            sql += " IN " + self.identifier_preparer.quote_identifier(schema)
+        return [row[1] for row in connection.exec_driver_sql(sql) if not 
row[2]]
+
+    def get_columns(self, connection, table_name, schema=None, **kw):
+        rows = connection.exec_driver_sql("DESCRIBE TABLE " + 
self._qualified(table_name, schema))
+        columns = []
+        for row in rows:
+            name = row[0]
+            # DESCRIBE separates metadata sections with rows that have no type.
+            # A real, quoted column can contain whitespace or start with '#'.
+            if not name or not row[1]:
+                break
+            columns.append({"name": name, "type": spark_type(row[1]), 
"nullable": True, "default": None})
+        return columns
+
+    def has_table(self, connection, table_name, schema=None, **kw):
+        if schema is None:
+            # Local temporary views are session-scoped and cannot be qualified
+            # with a database name, even though SHOW TABLES IN lists them.
+            if any(row[1] == table_name and row[2]
+                   for row in connection.exec_driver_sql("SHOW TABLES")):
+                return True
+        return (table_name in self.get_table_names(connection, schema=schema)
+                or table_name in self.get_view_names(connection, 
schema=schema))
+
+    def get_pk_constraint(self, connection, table_name, schema=None, **kw):
+        # Spark does not expose enforced SQL primary key constraints.
+        return {"constrained_columns": [], "name": None}
+
+    def get_foreign_keys(self, connection, table_name, schema=None, **kw):
+        return []
+
+    def get_indexes(self, connection, table_name, schema=None, **kw):
+        return []
+
+    def get_unique_constraints(self, connection, table_name, schema=None, 
**kw):
+        return []
+
+    def get_check_constraints(self, connection, table_name, schema=None, **kw):
+        return []
+
+    def get_table_comment(self, connection, table_name, schema=None, **kw):
+        return {"text": None}
diff --git 
a/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py 
b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
new file mode 100644
index 000000000000..2e34a2f74cf2
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Superset integration for Spark SQL over ADBC Connect."""
+
+from urllib.parse import quote, unquote
+
+from superset.db_engine_specs.base import BaseEngineSpec
+from superset.db_engine_specs.spark import time_grain_expressions
+from superset.sql.parse import SQLGLOT_DIALECTS
+
+# Superset maps SQL parser dialects by engine name.
+# This internal registry is tested with Superset 6.1.0; revalidate on upgrades.
+SQLGLOT_DIALECTS["sparkconnect"] = SQLGLOT_DIALECTS["spark"]
+
+
+class SparkConnectEngineSpec(BaseEngineSpec):
+    engine = "sparkconnect"
+    engine_name = "Apache Spark Connect (ADBC)"
+    supports_dynamic_schema = True
+    supports_file_upload = False
+    _time_grain_expressions = time_grain_expressions
+    metadata = {
+        "description": "Apache Spark SQL over Spark Connect using ADBC.",
+        "logo": "apache-spark.png",
+        "homepage_url": "https://spark.apache.org/sql/";,
+        "connection_string": 
"sparkconnect://{hostname}:{port}/{database}?api=connect",
+        "default_port": 15002,
+    }
+
+    @classmethod
+    def convert_dttm(cls, target_type, dttm, db_extra=None):
+        target = target_type.strip().upper()
+        if target == "DATE":
+            return f"CAST('{dttm.date().isoformat()}' AS DATE)"
+        if target in ("DATETIME", "TIMESTAMP", "TIMESTAMP_NTZ"):
+            value = dttm.isoformat(sep=" ", timespec="microseconds")
+            spark_type = "TIMESTAMP" if target == "DATETIME" else target
+            return f"CAST('{value}' AS {spark_type})"
+        return None
+
+    @classmethod
+    def epoch_to_dttm(cls):
+        return "TIMESTAMP_SECONDS({col})"
+
+    @classmethod
+    def epoch_ms_to_dttm(cls):
+        return "TIMESTAMP_MILLIS({col})"
+
+    @classmethod
+    def adjust_engine_params(cls, uri, connect_args, catalog=None, 
schema=None):
+        if catalog:
+            raise ValueError("Selecting a catalog is not supported")
+        if schema:
+            uri = uri.set(database=quote(schema, safe=""))
+        return uri, connect_args
+
+    @classmethod
+    def get_schema_from_engine_params(cls, sqlalchemy_uri, connect_args):
+        return unquote(sqlalchemy_uri.database) if sqlalchemy_uri.database 
else None
diff --git a/hudi-lakehouse/superset-spark-connect/pyproject.toml 
b/hudi-lakehouse/superset-spark-connect/pyproject.toml
new file mode 100644
index 000000000000..cbd9594fae5f
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/pyproject.toml
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hudi-spark-connect-superset"
+version = "0.1.0"
+requires-python = ">=3.9"
+dependencies = ["SQLAlchemy>=1.4,<2", "adbc-driver-manager==1.12.0", 
"pyarrow>=16.1,<19"]
+
+[project.entry-points."sqlalchemy.dialects"]
+sparkconnect = "hudi_spark_connect.dialect:SparkConnectDialect"
+
+[project.entry-points."superset.db_engine_specs"]
+sparkconnect = "hudi_spark_connect.engine_spec:SparkConnectEngineSpec"
+
+[tool.setuptools]
+packages = ["hudi_spark_connect"]
diff --git a/hudi-lakehouse/superset-spark-connect/test_dialect.py 
b/hudi-lakehouse/superset-spark-connect/test_dialect.py
new file mode 100644
index 000000000000..253676eeb06b
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_dialect.py
@@ -0,0 +1,137 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import unittest
+from unittest.mock import MagicMock, patch
+from urllib.parse import urlencode
+
+from sqlalchemy.engine.url import make_url
+from sqlalchemy import create_engine, types
+from sqlalchemy.dialects import registry
+from sqlalchemy import column, select
+from hudi_spark_connect.dialect import SparkConnectDialect, spark_type
+
+
+class DialectTest(unittest.TestCase):
+    def setUp(self):
+        self.dialect = SparkConnectDialect()
+
+    def test_connect_url_and_schema(self):
+        args, options = self.dialect.create_connect_args(make_url(
+            "sparkconnect://localhost:15002/a%60b?tls=false"))
+        self.assertEqual(args, [])
+        self.assertEqual(options["db_kwargs"]["uri"],
+                         
"spark://localhost:15002?tls=false&api=connect&auth_type=none")
+        self.dialect.dbapi = MagicMock()
+        conn = self.dialect.dbapi.connect.return_value
+        self.dialect.connect(*args, **options)
+        
conn.cursor.return_value.__enter__.return_value.execute.assert_called_once_with("USE
 `a``b`")
+        self.dialect.dbapi.connect.assert_called_once_with(
+            driver="spark", db_kwargs={"uri": 
"spark://localhost:15002?tls=false&api=connect&auth_type=none"}, 
autocommit=True)
+
+    def test_driver_uri_override_rejected(self):
+        for driver in ("spark", "other_manifest", "/tmp/x.so", "../x", 
r"C:\x.dll"):
+            with self.subTest(driver=driver), 
self.assertRaisesRegex(ValueError, "driver URI parameter"):
+                self.dialect.create_connect_args(make_url(
+                    "sparkconnect://localhost:15002/default?" + 
urlencode({"driver": driver})))
+
+    def test_engine_connect_args_cannot_override_driver(self):
+        from adbc_driver_manager import dbapi
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        for options in ({"driver": "/tmp/evil.so"}, {"driver": "other"}):
+            with self.subTest(options=options), patch.object(dbapi, "connect") 
as connect:
+                engine = create_engine("sparkconnect://localhost/default", 
connect_args=options, module=dbapi)
+                with self.assertRaises(ValueError):
+                    engine.raw_connection()
+                connect.assert_not_called()
+                engine.dispose()
+        with patch.object(dbapi, "connect", 
side_effect=RuntimeError("connection attempted")) as connect:
+            engine = create_engine("sparkconnect://localhost/default", 
module=dbapi)
+            with self.assertRaisesRegex(RuntimeError, "connection attempted"):
+                engine.raw_connection()
+            self.assertEqual(connect.call_args.kwargs["driver"], "spark")
+            engine.dispose()
+
+    def test_credentials_and_protocol_rejected(self):
+        for url in ("sparkconnect://u:p@host", 
"sparkconnect://host?api=thrift"):
+            with self.subTest(url=url), self.assertRaises(ValueError):
+                self.dialect.create_connect_args(make_url(url))
+
+    def test_failed_schema_selection_closes_connection(self):
+        self.dialect.dbapi = MagicMock()
+        conn = self.dialect.dbapi.connect.return_value
+        conn.cursor.return_value.__enter__.return_value.execute.side_effect = 
RuntimeError("bad schema")
+        with self.assertRaisesRegex(RuntimeError, "bad schema"):
+            self.dialect.connect(_schema="missing")
+        conn.close.assert_called_once()
+
+    def test_execute_never_binds_or_retries(self):
+        cursor = MagicMock()
+        self.dialect.do_execute(cursor, "SELECT 1", ())
+        cursor.execute.assert_called_once_with("SELECT 1")
+        with self.assertRaises(NotImplementedError):
+            self.dialect.do_execute(cursor, "SELECT ?", (1,))
+        with self.assertRaises(NotImplementedError):
+            self.dialect.do_executemany(cursor, "INSERT INTO t VALUES (?)", 
[(1,)])
+        self.assertEqual(cursor.execute.call_count, 1)
+
+    def test_offset_without_limit_does_not_add_negative_limit(self):
+        stmt = select(column("id")).offset(5)
+        sql = str(stmt.compile(dialect=self.dialect, 
compile_kwargs={"literal_binds": True}))
+        self.assertIn("OFFSET 5", sql)
+        self.assertNotIn("LIMIT", sql)
+        sql = str(stmt.limit(10).compile(
+            dialect=self.dialect, compile_kwargs={"literal_binds": True}))
+        self.assertIn("LIMIT 10", sql)
+        self.assertIn("OFFSET 5", sql)
+
+    def test_reflection_quoting_and_types(self):
+        conn = MagicMock()
+        conn.exec_driver_sql.side_effect = [
+            [("db", "v", False)], [("db", "t", False), ("db", "v", False), 
("db", "tmp", True)],
+            [("id", "int", ""), ("amount", "decimal(18, 2)", ""), ("", "", 
""), ("# Partition", "", "")],
+        ]
+        self.assertEqual(self.dialect.get_table_names(conn, 'a`b'), ["t"])
+        columns = self.dialect.get_columns(conn, "table", 'a`b')
+        self.assertEqual([x["name"] for x in columns], ["id", "amount"])
+        self.assertIsInstance(columns[0]["type"], types.Integer)
+        self.assertEqual((columns[1]["type"].precision, 
columns[1]["type"].scale), (18, 2))
+        self.assertIn("`a``b`.`table`", conn.exec_driver_sql.call_args.args[0])
+        self.assertIsInstance(spark_type("struct<a:int>"), types.NullType)
+
+
+    def test_reflection_preserves_quoted_column_names(self):
+        conn = MagicMock()
+        conn.exec_driver_sql.return_value = [
+            (" leading ", "int", ""), ("#metric", "string", ""),
+            ("", "", ""), ("# Partition Information", "", ""),
+        ]
+        columns = self.dialect.get_columns(conn, "t")
+        self.assertEqual([column["name"] for column in columns], [" leading ", 
"#metric"])
+
+    def test_has_table_finds_session_temporary_view(self):
+        conn = MagicMock()
+        conn.exec_driver_sql.return_value = [("", "temporary_view", True)]
+        self.assertTrue(self.dialect.has_table(conn, "temporary_view"))
+        conn.exec_driver_sql.assert_called_once_with("SHOW TABLES")
+
+    def test_has_table_does_not_bind_temporary_view_to_schema(self):
+        conn = MagicMock()
+        conn.exec_driver_sql.return_value = [("", "temporary_view", True)]
+        self.assertFalse(self.dialect.has_table(conn, "temporary_view", 
schema="other"))
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/hudi-lakehouse/superset-spark-connect/test_engine_spec.py 
b/hudi-lakehouse/superset-spark-connect/test_engine_spec.py
new file mode 100644
index 000000000000..eb0bfbc2a1c4
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_engine_spec.py
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run with Superset installed; exercises its actual filter handling path."""
+import importlib.util
+import unittest
+from datetime import datetime
+
+
[email protected](importlib.util.find_spec('superset'), 'requires Superset')
+class SupersetEngineSpecTest(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        from superset.app import create_app
+        cls.app = create_app()
+
+    def test_schema_round_trip(self):
+        from unittest.mock import MagicMock
+        from sqlalchemy.engine.url import make_url
+        from hudi_spark_connect.dialect import SparkConnectDialect
+        from hudi_spark_connect.engine_spec import SparkConnectEngineSpec as 
spec
+
+        for schema in ("a/b", "a`b", "a b", "a/b` c", "a%2Fb"):
+            with self.subTest(schema=schema):
+                uri, connect_args = spec.adjust_engine_params(
+                    
make_url("sparkconnect://localhost:15002/default?tls=false"), {}, schema=schema)
+                uri = make_url(str(uri))
+                self.assertEqual(spec.get_schema_from_engine_params(uri, 
connect_args), schema)
+                dialect = SparkConnectDialect()
+                args, options = dialect.create_connect_args(uri)
+                self.assertEqual(options["_schema"], schema)
+                dialect.dbapi = MagicMock()
+                dialect.connect(*args, **options)
+                cursor = 
dialect.dbapi.connect.return_value.cursor.return_value.__enter__.return_value
+                cursor.execute.assert_called_once_with("USE `" + 
schema.replace("`", "``") + "`")
+
+    def test_temporal_filter_values_compile(self):
+        import sqlalchemy as sa
+        from superset.models.helpers import ExploreMixin
+        from superset.utils.core import FilterOperator, GenericDataType
+        from hudi_spark_connect.engine_spec import SparkConnectEngineSpec
+
+        for native, expected in [('DATE', "CAST('2023-11-14' AS DATE)"),
+                                 ('DATETIME', "CAST('2023-11-14 
22:13:20.000000' AS TIMESTAMP)"),
+                                 ('TIMESTAMP', "CAST('2023-11-14 
22:13:20.000000' AS TIMESTAMP)"),
+                                 ('TIMESTAMP_NTZ', "CAST('2023-11-14 
22:13:20.000000' AS TIMESTAMP_NTZ)")]:
+            with self.subTest(native=native):
+                value = ExploreMixin.filter_values_handler(
+                    [1700000000000], FilterOperator.EQUALS, 
GenericDataType.TEMPORAL,
+                    target_native_type=native, 
db_engine_spec=SparkConnectEngineSpec)
+                sql = str(sa.select([sa.column('ts')]).where(sa.column('ts') 
== value).compile(
+                    compile_kwargs={'literal_binds': True}))
+                self.assertIn(expected, sql)
+
+    def test_epoch_expressions_and_unknown_type(self):
+        import sqlalchemy as sa
+        from hudi_spark_connect.engine_spec import SparkConnectEngineSpec as 
spec
+        self.assertEqual(spec.epoch_to_dttm(), 'TIMESTAMP_SECONDS({col})')
+        self.assertEqual(spec.epoch_ms_to_dttm(), 'TIMESTAMP_MILLIS({col})')
+        self.assertIsNone(spec.convert_dttm('STRING', datetime(2026, 1, 1)))
+        with self.app.app_context():
+            for fmt, function in [('epoch_s', 'TIMESTAMP_SECONDS'),
+                                  ('epoch_ms', 'TIMESTAMP_MILLIS')]:
+                with self.subTest(fmt=fmt):
+                    expr = spec.get_timestamp_expr(sa.column('ts'), fmt, 'P1D')
+                    sql = 
str(sa.select([expr]).compile(compile_kwargs={'literal_binds': True}))
+                    self.assertIn("date_trunc('day', " + function + '(ts))', 
sql)
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/hudi-lakehouse/superset-spark-connect/test_live.py 
b/hudi-lakehouse/superset-spark-connect/test_live.py
new file mode 100644
index 000000000000..ff270716eb56
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_live.py
@@ -0,0 +1,138 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in live test: creates and removes a unique database on the target 
server."""
+import os
+from datetime import datetime, timezone
+from decimal import Decimal
+import unittest
+import uuid
+
+from sqlalchemy import MetaData, Table, column, create_engine, inspect, 
select, text
+from sqlalchemy.dialects import registry
+
+URI = os.environ.get("HUDI_SUPERSET_CONNECT_TEST_URI")
+
+
[email protected](URI, "set HUDI_SUPERSET_CONNECT_TEST_URI to a disposable 
endpoint")
+class LiveDialectTest(unittest.TestCase):
+    def test_offset_only_compiled_sql(self):
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        engine = create_engine(URI)
+        try:
+            query = select(column("id")).select_from(text("range(10)"))\
+                .order_by(column("id")).offset(5)
+            sql = str(query.compile(dialect=engine.dialect, 
compile_kwargs={"literal_binds": True}))
+            with engine.connect() as conn:
+                self.assertEqual(conn.exec_driver_sql(sql).fetchall(), [(5,), 
(6,), (7,), (8,), (9,)])
+        finally:
+            engine.dispose()
+
+    def test_epoch_seconds_and_milliseconds(self):
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        engine = create_engine(URI)
+        try:
+            with engine.connect() as conn:
+                row = conn.exec_driver_sql(
+                    "SELECT unix_micros(timestamp_seconds(1700000000)), "
+                    "unix_micros(timestamp_millis(1700000000123)), "
+                    "unix_micros(timestamp_millis(-1))"
+                ).fetchone()
+                self.assertEqual(tuple(row), (1700000000000000, 
1700000000123000, -1000))
+        finally:
+            engine.dispose()
+
+    def test_hudi_read_write_and_reflection(self):
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        engine = create_engine(URI)
+        database = "connect_adapter_" + uuid.uuid4().hex
+        table = database + ".orders"
+        try:
+            with engine.connect() as conn:
+                conn.exec_driver_sql("CREATE DATABASE " + database)
+                try:
+                    conn.exec_driver_sql(f"""CREATE TABLE {table} (id INT, 
region STRING, amount INT, ts BIGINT, day DATE)
+                        USING hudi TBLPROPERTIES (type='cow', primaryKey='id', 
preCombineField='ts')""")
+                    conn.exec_driver_sql(f"INSERT INTO {table} VALUES (1, 
'East', 100, 1, DATE '2026-10-01'), (2, 'West', 200, 1, DATE '2026-10-01')")
+                    self.assertIn(database, inspect(conn).get_schema_names())
+                    self.assertIn('orders', 
inspect(conn).get_table_names(database))
+                    reflected = Table('orders', MetaData(), schema=database, 
autoload_with=conn)
+                    self.assertIn('amount', reflected.c)
+                    columns = inspect(conn).get_columns('orders', database)
+                    self.assertTrue({'id', 'region', 'amount', 'ts'} <= 
{c['name'] for c in columns})
+                    self.assertEqual(conn.exec_driver_sql(f"SELECT SUM(amount) 
FROM {table}").scalar(), 300)
+                    metadata = conn.exec_driver_sql(
+                        f"SELECT _hoodie_commit_time, _hoodie_record_key FROM 
{table} ORDER BY id"
+                    ).fetchall()
+                    self.assertEqual([row[1] for row in metadata], ['1', '2'])
+                    self.assertTrue(all(row[0] for row in metadata))
+                    self.assertEqual(conn.exec_driver_sql(
+                        f"SELECT region, SUM(amount) FROM {table} WHERE 
region='East' GROUP BY region"
+                    ).fetchall(), [('East', 100)])
+                    self.assertEqual(conn.exec_driver_sql(
+                        f"SELECT CAST(date_trunc('day', day) AS STRING), 
SUM(amount) "
+                        f"FROM {table} GROUP BY date_trunc('day', day)"
+                    ).fetchall(), [('2026-10-01 00:00:00', 300)])
+                    conn.exec_driver_sql(f"UPDATE {table} SET amount=150, ts=2 
WHERE id=1")
+                    self.assertEqual(conn.exec_driver_sql(f"SELECT id, amount 
FROM {table} ORDER BY id").fetchall(), [(1, 150), (2, 200)])
+                    conn.exec_driver_sql(f"DELETE FROM {table} WHERE id=2")
+                    self.assertEqual(conn.exec_driver_sql(f"SELECT id, amount 
FROM {table}").fetchall(), [(1, 150)])
+                finally:
+                    conn.exec_driver_sql(f"DROP TABLE IF EXISTS {table} PURGE")
+                    conn.exec_driver_sql(f"DROP DATABASE IF EXISTS {database}")
+        finally:
+            engine.dispose()
+
+    def test_session_view_and_quoted_columns(self):
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        engine = create_engine(URI)
+        name = "connect_view_" + uuid.uuid4().hex
+        try:
+            with engine.connect() as conn:
+                try:
+                    conn.exec_driver_sql(f"CREATE TEMP VIEW {name} AS SELECT 1 
AS ` leading `, 2 AS `#metric`")
+                    inspector = inspect(conn)
+                    self.assertTrue(inspector.has_table(name))
+                    self.assertFalse(inspector.has_table(name, 
schema="default"))
+                    self.assertEqual([column["name"] for column in 
inspector.get_columns(name)],
+                                     [" leading ", "#metric"])
+                    self.assertEqual(conn.exec_driver_sql(f"SELECT ` leading 
`, `#metric` FROM {name}").fetchall(), [(1, 2)])
+                finally:
+                    conn.exec_driver_sql(f"DROP VIEW IF EXISTS {name}")
+        finally:
+            engine.dispose()
+
+    def test_arrow_type_mapping(self):
+        registry.register("sparkconnect", "hudi_spark_connect.dialect", 
"SparkConnectDialect")
+        engine = create_engine(URI)
+        try:
+            with engine.connect() as conn:
+                with conn.connection.cursor() as cursor:
+                    cursor.execute("SET TIME ZONE 'UTC'")
+                    cursor.execute("""SELECT CAST('2026-01-02 03:04:05.123456' 
AS TIMESTAMP) AS ts,
+                        CAST('2026-01-02 03:04:05.123456' AS TIMESTAMP_NTZ) AS 
ntz,
+                        CAST(123.456 AS DECIMAL(38,18)) AS d,
+                        array(1, CAST(NULL AS INT)) AS a, named_struct('x', 7) 
AS s""")
+                    arrow = cursor.fetch_arrow_table()
+                    self.assertEqual(str(arrow.schema.field('ts').type), 
'timestamp[us, tz=UTC]')
+                    self.assertEqual(str(arrow.schema.field('ntz').type), 
'timestamp[us]')
+                    self.assertEqual(str(arrow.schema.field('d').type), 
'decimal128(38, 18)')
+                    self.assertEqual(arrow.to_pylist(), [{
+                        'ts': datetime(2026, 1, 2, 3, 4, 5, 123456, 
tzinfo=timezone.utc),
+                        'ntz': datetime(2026, 1, 2, 3, 4, 5, 123456),
+                        'd': Decimal('123.456000000000000000'),
+                        'a': [1, None], 's': {'x': 7}}])
+        finally:
+            engine.dispose()
diff --git a/hudi-lakehouse/superset-spark-connect/test_superset_live.py 
b/hudi-lakehouse/superset-spark-connect/test_superset_live.py
new file mode 100644
index 000000000000..e7106ae1ec9c
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_superset_live.py
@@ -0,0 +1,196 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in SQL Lab API test. The target connection must allow DML."""
+import os
+import unittest
+import uuid
+
+URL = os.environ.get("HUDI_SUPERSET_URL")
+USER = os.environ.get("HUDI_SUPERSET_USER")
+PASSWORD = os.environ.get("HUDI_SUPERSET_PASSWORD")
+DATABASE = os.environ.get("HUDI_SUPERSET_DATABASE_ID")
+SCHEMA = os.environ.get("HUDI_SUPERSET_SCHEMA", "default")
+
+
[email protected](all((URL, USER, PASSWORD, DATABASE)), "set Superset URL, 
user, password and database ID")
+class SupersetLiveTest(unittest.TestCase):
+    def test_temporal_chart_filters_and_epoch_grouping(self):
+        import requests
+
+        session = requests.Session()
+        base = URL.rstrip("/")
+        response = session.post(base + "/api/v1/security/login", json={
+            "username": USER, "password": PASSWORD, "provider": "db"}, 
timeout=30)
+        response.raise_for_status()
+        session.headers["Authorization"] = "Bearer " + 
response.json()["access_token"]
+        response = session.get(base + "/api/v1/security/csrf_token/", 
timeout=30)
+        response.raise_for_status()
+        session.headers.update({"X-CSRFToken": response.json()["result"], 
"Referer": base + "/"})
+        table = "temporal_adapter_" + uuid.uuid4().hex
+        dataset_id = None
+        created = False
+
+        def run(sql):
+            response = session.post(base + "/api/v1/sqllab/execute/", json={
+                "database_id": int(DATABASE), "sql": sql, "schema": SCHEMA,
+                "runAsync": False, "client_id": uuid.uuid4().hex[:11]}, 
timeout=180)
+            response.raise_for_status()
+            result = response.json()
+            self.assertEqual(result.get("status"), "success", result)
+            return result["data"]
+
+        def chart(query):
+            response = session.post(base + "/api/v1/chart/data", json={
+                "datasource": {"id": dataset_id, "type": "table"}, "queries": 
[query],
+                "result_format": "json", "result_type": "full", "force": 
True}, timeout=120)
+            response.raise_for_status()
+            result = response.json()["result"][0]
+            self.assertEqual(result["status"], "success", result)
+            return result
+
+        try:
+            run(f"CREATE TABLE {table} (id INT, day DATE, event_ts TIMESTAMP, "
+                "epoch_s BIGINT, epoch_ms BIGINT) USING parquet")
+            created = True
+            run(f"INSERT INTO {table} VALUES "
+                "(1, DATE '2023-11-14', TIMESTAMP '2023-11-14 22:13:20', 
1700000000, 1700000000000), "
+                "(2, DATE '2023-11-15', TIMESTAMP '2023-11-15 22:13:20', 
1700086400, 1700086400000)")
+            response = session.post(base + "/api/v1/dataset/", json={
+                "database": int(DATABASE), "schema": SCHEMA, "table_name": 
table}, timeout=120)
+            response.raise_for_status()
+            dataset_id = response.json()["id"]
+            for name in ("day", "event_ts"):
+                result = chart({"columns": ["id"], "filters": [
+                    {"col": name, "op": "==", "val": 1700000000000}], 
"row_limit": 100})
+                self.assertEqual(result["data"], [{"id": 1}])
+
+            response = session.get(base + f"/api/v1/dataset/{dataset_id}", 
timeout=30)
+            response.raise_for_status()
+            columns = response.json()["result"]["columns"]
+            for column in columns:
+                if column["column_name"] in ("epoch_s", "epoch_ms"):
+                    column["is_dttm"] = True
+                    column["python_date_format"] = column["column_name"]
+            keys = ("id", "column_name", "is_dttm", "python_date_format", 
"type", "groupby", "filterable")
+            response = session.put(base + f"/api/v1/dataset/{dataset_id}", 
json={
+                "columns": [{key: column[key] for key in keys} for column in 
columns]}, timeout=30)
+            response.raise_for_status()
+            for name in ("epoch_s", "epoch_ms"):
+                result = chart({"columns": [{"columnType": "BASE_AXIS", 
"sqlExpression": name,
+                                             "label": name, "timeGrain": 
"P1D"}],
+                                "metrics": [{"expressionType": "SQL", 
"sqlExpression": "COUNT(*)",
+                                             "label": "count"}], "row_limit": 
100})
+                self.assertEqual(sorted((row[name], row["count"]) for row in 
result["data"]),
+                                 [(1699920000000, 1), (1700006400000, 1)])
+            result = chart({"columns": ["id"], "orderby": [["id", True]],
+                            "row_offset": 1, "row_limit": None})
+            self.assertEqual(result["data"], [{"id": 2}])
+        finally:
+            try:
+                if dataset_id is not None:
+                    session.delete(base + f"/api/v1/dataset/{dataset_id}", 
timeout=30).raise_for_status()
+            finally:
+                if created:
+                    run(f"DROP TABLE IF EXISTS {table} PURGE")
+                session.close()
+
+    def test_sql_lab_queries_and_writes(self):
+        import requests
+
+        session = requests.Session()
+        base = URL.rstrip("/")
+        response = session.post(base + "/api/v1/security/login", json={
+            "username": USER, "password": PASSWORD, "provider": "db"}, 
timeout=30)
+        response.raise_for_status()
+        session.headers["Authorization"] = "Bearer " + 
response.json()["access_token"]
+        response = session.get(base + "/api/v1/security/csrf_token/", 
timeout=30)
+        response.raise_for_status()
+        session.headers.update({"X-CSRFToken": response.json()["result"], 
"Referer": base + "/"})
+        table = "connect_adapter_" + uuid.uuid4().hex
+
+        def run(sql):
+            response = session.post(base + "/api/v1/sqllab/execute/", json={
+                "database_id": int(DATABASE), "sql": sql, "schema": SCHEMA,
+                "runAsync": False, "client_id": uuid.uuid4().hex[:11]}, 
timeout=180)
+            response.raise_for_status()
+            result = response.json()
+            self.assertEqual(result.get("status"), "success", result)
+            return result["data"]
+
+        created = False
+        dataset_id = None
+        attempted = False
+        try:
+            attempted = True
+            run(f"CREATE TABLE {table} (id INT, region STRING, amount INT, ts 
BIGINT) USING hudi "
+                "TBLPROPERTIES(type='cow', primaryKey='id', 
preCombineField='ts')")
+            created = True
+            run(f"INSERT INTO {table} VALUES (1,'East',100,1), 
(2,'West',200,1)")
+            self.assertEqual(run(f"SELECT SUM(amount) AS total FROM {table}"), 
[{"total": 300}])
+            self.assertEqual(run(f"SELECT id FROM {table} WHERE 
region='East'"), [{"id": 1}])
+            response = session.post(base + "/api/v1/dataset/", json={
+                "database": int(DATABASE), "schema": SCHEMA, "table_name": 
table}, timeout=120)
+            response.raise_for_status()
+            dataset_id = response.json()["id"]
+            payload = {"datasource": {"id": dataset_id, "type": "table"},
+                       "queries": [{"metrics": [{"expressionType": "SQL",
+                           "sqlExpression": "SUM(amount)", "label": "total"}],
+                           "columns": ["region"], "row_limit": 100}],
+                       "result_format": "json", "result_type": "full", 
"force": True}
+
+            def chart_data():
+                response = session.post(base + "/api/v1/chart/data", 
json=payload, timeout=120)
+                response.raise_for_status()
+                result = response.json()["result"][0]
+                self.assertEqual(result["status"], "success", result)
+                return {row["region"]: row["total"] for row in result["data"]}
+
+            self.assertEqual(chart_data(), {"East": 100, "West": 200})
+            run(f"UPDATE {table} SET amount=150, ts=2 WHERE id=1")
+            self.assertEqual(run(f"SELECT id, amount FROM {table} WHERE 
id=1"), [{"id": 1, "amount": 150}])
+            self.assertEqual(chart_data(), {"East": 150, "West": 200})
+            run(f"MERGE INTO {table} t USING ("
+                "SELECT 1 AS id, 'East' AS region, 175 AS amount, CAST(3 AS 
BIGINT) AS ts "
+                "UNION ALL SELECT 3, 'North', 300, CAST(3 AS BIGINT)) s "
+                "ON t.id=s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED 
THEN INSERT *")
+            self.assertEqual(run(f"SELECT id, region, amount, ts FROM {table} 
ORDER BY id"), [
+                {"id": 1, "region": "East", "amount": 175, "ts": 3},
+                {"id": 2, "region": "West", "amount": 200, "ts": 1},
+                {"id": 3, "region": "North", "amount": 300, "ts": 3},
+            ])
+            run(f"DELETE FROM {table} WHERE id=2")
+            self.assertEqual(run(f"SELECT count(*) AS n FROM {table}"), [{"n": 
2}])
+        finally:
+            try:
+                if dataset_id is not None:
+                    try:
+                        response = session.delete(base + 
f"/api/v1/dataset/{dataset_id}", timeout=30)
+                        response.raise_for_status()
+                    except Exception:
+                        print(f"Cleanup failed; remove test dataset 
{dataset_id} manually.")
+                        raise
+            finally:
+                try:
+                    if created:
+                        try:
+                            run(f"DROP TABLE IF EXISTS {table} PURGE")
+                        except Exception:
+                            print(f"Cleanup failed; remove test table 
{SCHEMA}.{table} manually.")
+                            raise
+                    elif attempted:
+                        print(f"CREATE did not report success; check for test 
table {SCHEMA}.{table} before cleanup.")
+                finally:
+                    session.close()
diff --git a/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py 
b/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
new file mode 100644
index 000000000000..57e3b12a936b
--- /dev/null
+++ b/hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Opt-in Zeppelin notebook test against an isolated Connect endpoint."""
+import os
+import unittest
+import uuid
+from urllib.parse import parse_qsl, urlsplit
+
+URL = os.environ.get('HUDI_ZEPPELIN_URL')
+CONNECT_URI = os.environ.get('HUDI_ZEPPELIN_CONNECT_URI')
+
+
[email protected](URL and CONNECT_URI, 'set HUDI_ZEPPELIN_URL and 
HUDI_ZEPPELIN_CONNECT_URI')
+class ZeppelinLiveTest(unittest.TestCase):
+    def test_notebook_crud_and_table_output(self):
+        import requests
+
+        parsed = urlsplit(CONNECT_URI)
+        options = dict(parse_qsl(parsed.query, keep_blank_values=True))
+        allowed = {'api', 'auth_type', 'tls', 'validateservercertificate'}
+        if (parsed.scheme != 'spark' or parsed.username or parsed.password
+                or options.get('auth_type') != 'none' or options.get('api') != 
'connect'
+                or set(options) - allowed):
+            self.fail('Use an unauthenticated test URI without credentials: it 
is stored in the notebook')
+        base = URL.rstrip('/')
+        table = 'zeppelin_test_' + uuid.uuid4().hex
+        name = 'Hudi Connect test ' + uuid.uuid4().hex
+        # The URI is rendered as a Python string, not interpolated into SQL.
+        code = f'''%python
+from adbc_driver_manager import dbapi
+import pandas as pd
+with dbapi.connect(driver="spark", db_kwargs={{"uri": {CONNECT_URI!r}}}, 
autocommit=True) as conn:
+    with conn.cursor() as cur:
+        cur.execute("CREATE TABLE default.{table} (id INT, region STRING, 
amount INT, ts BIGINT) USING hudi TBLPROPERTIES(type='cow', primaryKey='id', 
preCombineField='ts')")
+        try:
+            cur.execute("INSERT INTO default.{table} VALUES (1,'East',100,1), 
(2,'West',200,1)")
+            cur.execute("SELECT region, SUM(amount) FROM default.{table} GROUP 
BY region ORDER BY region")
+            assert cur.fetchall() == [('East',100),('West',200)]
+            cur.execute("UPDATE default.{table} SET amount=150, ts=2 WHERE 
id=1")
+            cur.execute("SELECT id, amount FROM default.{table} WHERE id=1")
+            assert cur.fetchall() == [(1,150)]
+            cur.execute("MERGE INTO default.{table} t USING (SELECT 1 AS id, 
'East' AS region, 175 AS amount, CAST(3 AS BIGINT) AS ts UNION ALL SELECT 3, 
'North', 300, CAST(3 AS BIGINT)) s ON t.id=s.id WHEN MATCHED THEN UPDATE SET * 
WHEN NOT MATCHED THEN INSERT *")
+            cur.execute("SELECT id, region, amount, ts FROM default.{table} 
ORDER BY id")
+            assert cur.fetchall() == 
[(1,'East',175,3),(2,'West',200,1),(3,'North',300,3)]
+            cur.execute("DELETE FROM default.{table} WHERE id=2")
+            cur.execute("SELECT region, SUM(amount) FROM default.{table} GROUP 
BY region ORDER BY region")
+            rows = cur.fetchall()
+            assert rows == [('East',175),('North',300)]
+            z.show(pd.DataFrame(rows, columns=['region','total']))
+        finally:
+            cur.execute("DROP TABLE IF EXISTS default.{table} PURGE")
+'''
+        response = requests.post(base + '/api/notebook', json={'name': name, 
'paragraphs': [{
+            'title': 'ADBC Hudi CRUD', 'text': code,
+            'config': {'colWidth': 12, 'results': [{'graph': {'mode': 
'table'}}]}}]}, timeout=30)
+        response.raise_for_status()
+        note = response.json()['body']
+        succeeded = False
+        try:
+            response = requests.get(base + '/api/notebook/' + note, timeout=30)
+            response.raise_for_status()
+            paragraph = response.json()['body']['paragraphs'][0]['id']
+            response = requests.post(base + 
f'/api/notebook/run/{note}/{paragraph}', timeout=240)
+            response.raise_for_status()
+            result = response.json()['body']
+            self.assertEqual(result['code'], 'SUCCESS', result)
+            self.assertTrue(any(item['type'] == 'TABLE' and item['data'] == 
'region\ttotal\nEast\t175\nNorth\t300\n'
+                                for item in result['msg']), result)
+            succeeded = True
+        finally:
+            try:
+                response = requests.delete(base + '/api/notebook/' + note, 
timeout=30)
+                response.raise_for_status()
+            finally:
+                if not succeeded:
+                    print(f'Check test table default.{table} after the 
notebook stops before cleaning it up.')
+
+
+if __name__ == '__main__':
+    unittest.main()

Reply via email to