morningman opened a new issue, #67578:
URL: https://github.com/apache/doris/issues/67578
This issue tracks two halves of one goal: bringing Doris's **Arrow Flight
SQL server up to the
spec's feature set**, and maintaining an **official `doris` driver in the
ADBC Driver Foundry**, so
that a user runs `dbc install doris` and connects with a single URI.
Part of #65615. Part A depends on #67577 (one protocol-agnostic session and
execution layer) for its
session and prepared-statement items; Part B can start before that and ship
a pre-release built on
the subset that works today.
## Why
ADBC is becoming for columnar analytics what JDBC is for row-oriented
access. The ADBC Driver
Foundry already hosts drivers for ClickHouse, Databricks, Snowflake, Trino,
Spark, SingleStore,
Presto, Redshift, MSSQL and BigQuery, all implementing ADBC 1.1 and callable
from Go, Python, R and
Rust. Doris is not on that list, so when someone picks a data source in
pandas, polars, Ibis or R,
Doris is not among the options.
Doris users are not blocked today - they use the generic
`adbc_driver_flightsql` and then deal with
the gaps themselves. Doris's own FAQ lists twelve of them: BE unreachable
from the client unless
`public_host` or an Nginx reverse proxy is configured, tokens evicted when a
client never calls
`close()`, DATETIME arriving as an integer on the Java side, nested ARRAY
failing in JDBC clients,
gRPC message-size limits, `parameter ordinal 1 out of range` on
prepared-statement binding. A driver
is supposed to absorb that.
At the same time the server implements only a small part of the spec.
Session actions, parallel
endpoints, native metadata commands, bulk ingestion, cancellation and
renewal are all missing, so
users emulate them with SQL strings - `SET exec_mem_limit=2000` in place of
a session option.
**Design principles.** One ADBC connection is one Doris session is one
Flight SQL connection: no
second channel and no MySQL-protocol fallback, because a fallback path hides
missing capabilities
instead of reporting them - a gap should surface as
`ADBC_STATUS_NOT_IMPLEMENTED` and appear in the
capability matrix. Only the spec, no private RPCs: Doris-specific semantics
ride on session-option
names and Arrow extension type names, which are the extension points the
spec already provides. The
control plane stays in the FE and the data plane in the BE, but both belong
to one session - session
state only affects planning, and after planning the FE issues tickets and
the BE streams, so a
distributed data plane does not break the session's singularity.
## Status convention
Same as #65615: `[x]` means merged or confirmed complete; `[ ]` means open
or needs follow-up.
## Part A - server-side completeness
- [ ] **A1. Session lifecycle through the standard actions.**
`SetSessionOptions` /
`GetSessionOptions` / `CloseSession` mapped onto Doris session variables
(`SET`, `SHOW VARIABLES`),
with `adbc.connection.current_catalog` and `current_db_schema` as session
options (`SWITCH`,
`USE`). `CloseSession` also invalidates the session's authentication
context, as the spec requires.
The session object is the one from #67577, held by the Flight adapter,
which removes the separate
token cache and the `invalid bearer token` failure mode with it. Covers
#59742 and the client-close
half of #36331.
- [ ] **A2. Parallel result return.** `GetFlightInfo` returns N
`FlightEndpoint`s, one per BE result
sink, each with its own ticket, fetched concurrently. Three shapes:
scan-only queries grouped by
tablet (row bandwidth equals cluster bandwidth); aggregation or join at
the top, one sink per
fragment instance (what `enable_parallel_result_sink` already does - this
proposes making it the
default rather than an opt-in); a global sort at the top, still multiple
endpoints but with
`FlightInfo.ordered = true` and endpoint order equal to result order, so
clients fetch in parallel
and concatenate. Tickets must be self-contained and verifiable (query id,
fragment instance id,
expiry, signature) so the BE never has to call back to the FE, which would
make the FE the new
bottleneck. BE addressing uses the spec's
`arrow-flight-reuse-connection://?` location, which is a
cleaner and standard answer to the reachability problem behind #62538,
#62217, #59490 and #44599.
Related: #67368, where the short-circuit point-query path registers no
Flight endpoint at all.
- [ ] **A3. Native metadata commands.** `CommandGetCatalogs`,
`GetDbSchemas`, `GetTables`,
`GetTableTypes`, `GetXdbcTypeInfo`, `GetPrimaryKeys`, `GetImportedKeys`,
`GetExportedKeys`,
`GetSqlInfo` served from catalog metadata directly as Arrow, instead of
being rewritten into
`information_schema` SQL. Doris's multi-catalog model (internal plus Hive,
Iceberg, JDBC, Paimon)
maps exactly onto ADBC's catalog / db_schema / table hierarchy, so one
ADBC connection gives a
Python client a federated view of the whole lakehouse - something the
Snowflake and ClickHouse
drivers cannot offer.
- [ ] **A4. Bulk ingestion through `CommandStatementIngest`.** ADBC's
`adbc.ingest.target_table` plus
`BindStream` plus `ExecuteUpdate` is `DoPut` carrying
`CommandStatementIngest`, with
`DoPutUpdateResult` returned in `PutResult.app_metadata`. Two pieces on
the Doris side: an Arrow IPC
input format for Stream Load, and routing `CommandStatementIngest` to it,
so `conn.adbc_ingest()`
lands Arrow memory without a CSV or JSON round trip. The FE should
redirect the ingest `DoPut` to
the BEs the way Stream Load already redirects, and allow parallel writes,
otherwise ingestion
becomes the new bottleneck. Ingest modes map to `CREATE TABLE` + load,
Stream Load,
`INSERT OVERWRITE`, and create-then-append. Covers the ingestion request
in #25514.
- [ ] **A5. Prepared statements and parameter binding.**
`ActionCreatePreparedStatementRequest`
returns an opaque handle plus, optionally, result and parameter schemas;
per the spec the result
schema may depend on the bound parameters, so it is better to return
nothing than something wrong.
Doris reuses the prepared-statement object extracted in #67577 and
receives Arrow parameter batches
over `DoPut`. `ExecuteSchema` needs plan-without-execute, which the FE
planner can already do.
Covers
[apache/arrow-java#1009](https://github.com/apache/arrow-java/issues/1009); the
`parameter ordinal 1 out of range` error itself is the upstream bug
[apache/arrow#40118](https://github.com/apache/arrow/issues/40118) and
needs tracking there.
- [ ] **A6. Cancellation, renewal and polling.** `AdbcStatementCancel` onto
`CancelFlightInfo` with
the query id taken from `FlightInfo.app_metadata`, cancelling the
associated poll descriptors and
any issued-but-unconsumed tickets; `RenewFlightEndpoint` to extend the
life of a large result;
`PollFlightInfo` for asynchronous submission and progress of long queries.
- [ ] **A7. Authentication and compression.** Bearer token from `Handshake`
as the session credential,
mTLS, and the existing LDAP and Kerberos plugins; Arrow IPC body
compression (LZ4, ZSTD) negotiated
as a connection option, which matters most across data centers.
## Part B - the official driver
- [ ] **B1. Repository and stack.** An `adbc-drivers/doris` repository under
the Foundry (open an
issue in `adbc-drivers/onboarding` from the template; Foundry admins
create the repo with CI,
workflows and validation templates, then the Doris community members are
invited). Written in Go on
`driverbase-go`, built as a shared library exporting `AdbcDriverDorisInit`.
- [ ] **B2. Layering.** `driver/` (ADBC surface), `flight/` (handshake,
token renewal, endpoint
routing and parallel scheduling, retries), `types/` (type mapping and
Arrow extension registration),
`catalog/` (metadata commands to `GetObjects`), `ingest/`. The type
mapping in `types/` shares one
source of truth with the server via code generation, so the two cannot
drift.
- [ ] **B3. ADBC 1.1 surface** mapped onto Flight SQL calls, including
`ExecutePartitions` returning
serialized `FlightEndpoint`s as partitions and `ReadPartition` opening one
`DoGet` each, with
default parallelism `min(len(endpoints), GOMAXPROCS)`.
- [ ] **B4. Driver options and URI.** `doris.read.parallelism`,
`doris.read.compression`,
`doris.grpc.max_message_size`, `doris.ingest.format`,
`doris.catalog.include_external`, and a single
URI form `doris://user:pass@fe_host:port/catalog.db?param=value` - no more
`grpc://` in Python
versus `jdbc:arrow-flight-sql://` in Java. The initial catalog/schema in
the URI covers
[apache/arrow-java#829](https://github.com/apache/arrow-java/issues/829).
## Testing
- [ ] **Baseline first, before any code.** Install the Foundry `mysql`
driver, point it at Doris's
MySQL port, run the Foundry validation suite and record what fails. That
list is both the
requirements document and good community-issue material. (The Foundry
`mysql` driver already
documents a non-MySQL backend, Databend, through its `mysql.vendor`
option, so this is a supported
way to measure.)
- [ ] A `DriverQuirks` implementation declaring Doris's capabilities, a
`docs/doris.md` skeleton
filled in from validation results, and a `compose.yaml` so validation runs
against a real cluster in
CI on every PR and release.
- [ ] Parallel-read correctness in the Doris repo: merged multi-endpoint
results equal the
single-endpoint result, `ordered = true` really is ordered, and partial
endpoint failure propagates
the error as expected. The cross-protocol consistency suite from #67577 is
reused as is.
- [ ] Throughput and memory baselines on a ClickBench or TPC-H result-fetch,
comparing `jdbc:mysql`,
the generic flightsql driver and the official driver.
## Release and promotion
The Foundry's bar for a first release is Linux, macOS and Windows, the three
minimum capabilities
(query, bulk ingest, `GetObjects`), and a passing standard validation run;
anything short of that
ships as a pre-release such as `v0.1.0-alpha.1`.
Suggested sequence: Part B ships an alpha on the subset that works today,
the validation report then
drives Part A's priorities, and each finished Part A item removes one
`NOT_IMPLEMENTED` from the
driver. When the minimum set is complete, tag v0.1.0 and ask the Foundry
admins for CDN
distribution.
Arrow Flight SQL is still marked experimental in Doris, which is the right
window for the breaking
changes in A1 (session and token lifetime) and A2 (parallel return changes
result arrival order for
unordered queries). Both go in the release notes. Proposed condition for
promoting Flight SQL out of
experimental: **A1 through A6 complete and the Foundry validation minimum
set green.**
The generic `adbc_driver_flightsql` keeps working and is not deprecated; the
official driver is the
recommended path, not the only one.
## Rejected alternatives
- **Rename the generic flightsql driver and publish it under the Foundry.**
Fastest way onto the
driver list, but it solves none of the user-visible problems - the
connection string, BE
reachability and type conversion all stay with the user - and replacing it
later costs more than
doing it properly once.
- **Wrap `go-sql-driver/mysql` in sqlwrapper and speak the MySQL protocol.**
Zero kernel changes and
compatible with every version, but the Arrow conversion happens on the
client, so it unifies the
API without gaining any transport speedup - which was the reason for doing
this at all. Valuable as
the measurement baseline above, not as the destination.
- **Two channels on the client: control plane over MySQL, data plane over
Flight SQL.** The session is
then not shared; see the same entry in #67577.
- **Private Flight extension RPCs for Doris-specific semantics.** Session
options and Arrow extension
types already cover it, and private RPCs would make the driver unusable by
generic clients, which
defeats the point of shipping a standard driver.
## References
- Arrow Flight SQL spec: https://arrow.apache.org/docs/format/FlightSql.html
- Building a driver in the ADBC Driver Foundry:
https://adbc-drivers.org/building-drivers/
- Existing Foundry drivers: https://adbc-drivers.org/drivers/
- `driverbase-go`: https://github.com/adbc-drivers/driverbase-go
- Foundry validation suite: https://github.com/adbc-drivers/validation
- Doris Arrow Flight SQL docs:
https://doris.apache.org/docs/dev/connection-integration/arrow-flight-sql
- The original Arrow Flight SQL issue: #25514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
