wenzhenghu opened a new issue, #67365: URL: https://github.com/apache/doris/issues/67365
### Search before asking - [x] I searched the existing issues and found no similar issue. ### Version Verified against the following upstream heads on 2026-09-01: - `branch-4.0`: `8a9961723ea4be00cdf923c60759607202c7e2e7` - `branch-4.1`: `6f4c6a4be42ab3f4e1811982703d4b71b5a8ea3c` - `master`: `feb9e04f78490296c3393cbd594aef617af6b433` Related preload PRs: - #64035 (`master`) - #64579 (`branch-4.1` backport) ### What is wrong? JDBC external tables do not have a consistent, cache-safe failure contract when remote metadata returns no columns or the remote table handle can no longer be resolved. #### branch-4.0 and branch-4.1 The failure chain is: 1. `JdbcExternalTable.initSchema()` returns `Optional.empty()` when `listColumns()` returns `null` or an empty list. 2. The external schema cache stores the negative result. 3. `ExternalTable.getFullSchema()` maps the empty value to `null`. 4. `getBaseSchema()` also returns `null`. 5. `LogicalCatalogRelation.computeOutput()` calls `table.getBaseSchema().stream()` and throws a null pointer exception. Typical error: ```text Cannot invoke "java.util.List.stream()" because the return value of "org.apache.doris.catalog.TableIf.getBaseSchema()" is null ``` `branch-4.1` additionally supports JDBC metadata preload through #64579. When `enable_preload_external_metadata=true` and a mixed query contains both an internal table requiring a plan-time read lock and a JDBC table, `PreloadExternalMetadata` calls `getBaseSchema()` before locking but does not validate its result. Analysis later reaches the same `stream()` NPE. Disabling preload does not remove the underlying bug; it only changes when the schema is loaded. `branch-4.0` does not contain the preload feature, but regular JDBC query planning still has the same nullable-schema chain. Relevant code: - branch-4.0 JDBC loader: https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java#L129-L136 - branch-4.0 nullable schema: https://github.com/apache/doris/blob/8a9961723ea4be00cdf923c60759607202c7e2e7/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java#L175-L184 - branch-4.1 JDBC loader: https://github.com/apache/doris/blob/6f4c6a4be42ab3f4e1811982703d4b71b5a8ea3c/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalTable.java#L130-L157 - branch-4.1 preload call: https://github.com/apache/doris/blob/6f4c6a4be42ab3f4e1811982703d4b71b5a8ea3c/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/PreloadExternalMetadata.java#L102-L111 - nullable consumer: https://github.com/apache/doris/blob/6f4c6a4be42ab3f4e1811982703d4b71b5a8ea3c/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java#L134-L141 #### master `master` has migrated JDBC catalogs to `PluginDrivenExternalTable` and therefore needs a separate fix rather than a mechanical backport. - `PluginDrivenExternalTable.initSchema()` returns `Optional.empty()` when the connector table handle is missing. - With a valid handle but zero returned columns, it creates a present schema cache value containing an empty column list. - `ExternalTable.getFullSchema()` still maps an empty optional to `null`, while consumers such as `LogicalCatalogRelation` assume a non-null list. The latest metadata-cache refactor in #66633 changes the common default path: the cache loader now converts `Optional.empty()` into a generic `CacheException` before `getFullSchema()` returns `null`. This normally avoids the exact NPE, but it is not an equivalent JDBC fix: - the error does not include actionable JDBC catalog and remote table context; - failed loads are completed exceptionally and are not retained as negative cache entries; - repeated queries can repeatedly execute remote table-handle or metadata resolution; - zero-column schemas with a valid handle are still represented as a normal empty schema. Relevant code: - plugin-driven schema loader: https://github.com/apache/doris/blob/feb9e04f78490296c3393cbd594aef617af6b433/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java#L459-L513 - nullable base implementation: https://github.com/apache/doris/blob/feb9e04f78490296c3393cbd594aef617af6b433/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java#L180-L195 - generic cache exception: https://github.com/apache/doris/blob/feb9e04f78490296c3393cbd594aef617af6b433/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java#L524-L535 - non-null consumer assumption: https://github.com/apache/doris/blob/feb9e04f78490296c3393cbd594aef617af6b433/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java#L152-L159 ### What did you expect? When JDBC schema metadata cannot be resolved: 1. Query analysis should fail with a deterministic, actionable exception containing the catalog and remote database/table name. 2. No planner path should expose a `null` schema to callers that require a schema list. 3. A persistent metadata failure should not cause every repeated query to reconnect to or reload metadata from the remote JDBC source. 4. Preload-enabled and preload-disabled queries should have the same error semantics. 5. The generic preload rule should not invalidate the table schema cache on every failure. ### How to reproduce Possible reproduction conditions include a JDBC driver returning no rows from `DatabaseMetaData.getColumns()`, insufficient metadata permission, or a remote table being removed while Doris still has a table-name entry. For `branch-4.1`: 1. Create a JDBC catalog and make one remote table visible through table listing. 2. Make the column metadata lookup return an empty result. 3. Enable `enable_preload_external_metadata`. 4. Run a Nereids mixed query joining an internal Doris table and the JDBC table. 5. Observe that preload reads the empty schema and later planning throws the `getBaseSchema().stream()` NPE. 6. Repeat with preload disabled; the same nullable schema can fail during normal relation output computation. For `branch-4.0`, run the JDBC query without the preload-specific steps. For `master`, make `JdbcConnectorMetadata.getTableHandle()` return empty, or return a valid handle with an empty `ConnectorTableSchema`, then repeat the query and inspect both the exception and the number of remote metadata calls. ### Suggested fix and tests For `branch-4.0` and `branch-4.1`: - preserve `Optional.empty()` in the existing schema cache as a negative entry; - reject the missing schema at the JDBC table schema-consumption boundary with an actionable `JdbcClientException`; - cover both `getFullSchema()` and inherited `getBaseSchema()` paths; - keep `PreloadExternalMetadata` generic and unchanged. For `master`, adapt the same contract to `PluginDrivenExternalTable` and the unified metadata cache, preferably with an explicit missing-schema result or bounded negative-cache representation rather than uncached loader exceptions. Tests should cover: - null and empty JDBC column metadata; - missing connector table handle; - preload enabled and disabled; - JDBC-only and mixed internal/JDBC queries; - repeated reads do not repeat remote metadata access while a negative entry is valid; - normal non-empty schemas remain unchanged. ### Anything else? This report distinguishes the exact NPE on `branch-4.x` from the current `master` behavior after #66633. The underlying cross-version issue is the lack of a consistent JDBC missing-schema contract and bounded negative caching. ### Are you willing to submit PR? - [ ] Yes, I am willing to submit a PR. ### Code of Conduct - [x] I agree to follow this projects Code of Conduct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
