Hi all, I'd like to propose adding two new SqlInfo codes to FlightSql.proto so that a server can advertise which part of Substrait it accepts:
- SQL_SUBSTRAIT_DIALECT (577): a UTF-8 string holding a Substrait dialect, i.e. a YAML document listing the types, relations, expressions and function signatures the server accepts in a plan - SQL_SUBSTRAIT_DIALECT_HASH (578): an opaque string identifying the dialect's content, for client-side caching Motivation. Flight SQL can already say whether a server accepts Substrait (FLIGHT_SQL_SERVER_SUBSTRAIT) and which versions (_MIN_VERSION / _MAX_VERSION), but not which part of the spec. In practice no consumer implements all of Substrait, so a producer only finds out a plan is unsupported when execution fails. Substrait now has a standard format for describing this: dialect files ( https://github.com/substrait-io/substrait/blob/main/text/dialect_schema.yaml). substrait-java already publishes one for its Spark integration ( https://github.com/substrait-io/substrait-java/blob/main/spark/spark_dialect.yaml ). Why a separate hash code. Dialects aren't tiny. One listing every function impl in the standard Substrait extensions (207 functions, 507 impls) plus all types, relations and expressions is about 34 KiB / ~1,900 lines of YAML. Real dialects are subsets of that. That's fine to return once per connection, but a chatty client can fetch only the hash and refetch the dialect when it changes. The hash is opaque and compared for equality only; the server must return a new value whenever the dialect changes (a hex-encoded SHA-256 of the dialect is suggested, not required). Semantics. The dialect should only be returned if FLIGHT_SQL_SERVER_SUBSTRAIT is true. If a server doesn't return it, the server doesn't describe which parts of Substrait it accepts; that doesn't mean it accepts all of Substrait. Placement. I've put both codes in the [500, 1000) range next to the other SQL syntax codes, since they describe the syntax a server accepts. That also means the change applies to ADBC, which reuses Flight SQL's SqlInfo codes. If people would rather see them next to the FLIGHT_SQL_SERVER_SUBSTRAIT* codes in the server info range, I can move them, but ADBC would then need a separate change. Changes are fully backwards-compatible: no existing codes or types change. - Issue: https://github.com/apache/arrow/issues/53872 - PR: https://github.com/apache/arrow/pull/54360 Thanks, Tornike
