xiangge1208 opened a new issue, #67484: URL: https://github.com/apache/doris/issues/67484
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description I would like to request adding regular expression (or wildcard) support for the `exclude_tables` and `include_tables` configurations in `CREATE JOB ... ON STREAMING FROM MYSQL`. Currently, `exclude_tables` / `include_tables` perform exact string matching only: `fe/fe-core/src/main/java/org/apache/doris/job/util/StreamingJobUtils.java` ```java excludeTablesList = Arrays.asList(excludeTables.split(",")); // L378-380 // ... if (includeTablesList.isEmpty() && !excludeTablesList.isEmpty() && excludeTablesList.contains(table)) { // L401-402 continue; } ``` ### Solution Support regex (or at minimum glob-style wildcards) in both `exclude_tables` and `include_tables`. Flink CDC's `tables` option already accepts regex (e.g. `app_db.\.*`), so this would also align the two code paths. Suggested semantics: - Keep backward compatibility: a pattern with no regex metacharacters behaves exactly as today. - Match against the bare table name (as today), not the qualified name, to avoid ambiguity. - Document whether the match is full-match or partial-match (`matches()` vs `find()`) — full-match is less surprising. Two related nits found while investigating, worth fixing in the same change: 1. **No trimming.** `excludeTables.split(",")` keeps surrounding whitespace, so `"a, b"` silently fails to exclude `b`. A `.trim()` per element would prevent a class of hard-to-spot config errors. 2. **Error ordering.** `generateCreateTableCmds` collects `noPrimaryKeyTables` while iterating, but throws only *after* the loop (`StreamingJobUtils.java` L469-472). Tables processed before the failure have already been created in Doris, so a failed `CREATE JOB` leaves partial state that must be cleaned up manually. Failing fast, or rolling back created tables, would be friendlier. ### Use case For whole-database sync this becomes hard to maintain. In our production database (285 base tables) there are 16+ manually-created backup/archive tables following recognizable naming patterns: ``` sys_config_bak_20260730 sys_config_bak_20260803_codex sys_config_bak_20260804 sys_config_bak_codex_progress_20260810 sys_config_bak_mcp_20260812 sys_config_bak_progress_20260813 sys_config_bak_progress_202608132 sys_config_bak_xie_20260806 sys_config_backup_20260710_codex_mcp_skill_sync_fix sys_config_backup_20260710_codex_mcp_workflow plugin_config_backup_20260721 sys_credit_record_archive_bak_20260805 sys_dept_permission_scope_bak_20260509 sys_permission_bak_20260808 sys_permission_model_bak_20260820 sys_role_bak_20260808 ``` With exact matching, every one of these must be enumerated, and — more importantly — **new backup tables created upstream after the job was created are silently picked up**. Since `exclude_tables` cannot be changed by `ALTER JOB`, keeping the exclusion list current requires dropping and recreating the job (and carefully preserving the binlog offset). A pattern like `sys_config_bak_.*` or `*_bak_*` would express the intent once and stay correct. ### Related issues _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
