nzw921rx opened a new issue, #11007: URL: https://github.com/apache/seatunnel/issues/11007
We need community help to migrate connector and transform validation from imperative `if/throw` checks to declarative `optionRule()` + `Conditions.*`. If the `Issue` column shows an existing issue number, please claim in that issue. If the `Issue` column shows `This issue`, please comment here to claim it. ## Prerequisites / Framework Status Phase 1 (framework) has been completed: - Design and scope audit: https://github.com/apache/seatunnel/issues/10976 - Framework implementation: https://github.com/apache/seatunnel/pull/10977 This umbrella issue is for **Phase 2 migration and tracking** only. ## Migration Guide ### A. Numeric range Use declarative constraints for rules like `port > 0`, `batchSize >= 0`. ```java import static org.apache.seatunnel.api.configuration.util.Conditions.greaterThan; OptionRule.builder() .required(PORT, greaterThan(PORT, 0)) .build(); ``` ### B1. Required cross-field comparison Use when both fields are mandatory and have relation constraints. ```java import static org.apache.seatunnel.api.configuration.util.Conditions.lessThanField; OptionRule.builder() .required(START_TIMESTAMP, END_TIMESTAMP, lessThanField(START_TIMESTAMP, END_TIMESTAMP)) .build(); ``` ### B2. Optional cross-field comparison Use when fields are not mandatory, but must satisfy relation when present. ```java import static org.apache.seatunnel.api.configuration.util.Conditions.lessOrEqualField; OptionRule.builder() .optional(MIN_VALUE, MAX_VALUE, lessOrEqualField(MIN_VALUE, MAX_VALUE)) .build(); ``` Do not accidentally convert optional semantics into required semantics. ### C. Conditional value check Use when a trigger option enables another constraint. ```java import static org.apache.seatunnel.api.configuration.util.Conditions.greaterThan; OptionRule.builder() .conditional( IGNORE_NO_LEADER_PARTITION, true, greaterThan(PARTITION_DISCOVERY_INTERVAL_MILLIS, 0)) .build(); ``` ### D1. Deprecated key coexistence (`withFallbackKeys` → `exclusive`) When migrating a deprecated key to a new key name, replace `withFallbackKeys` + runtime `sourceMap` mutual-exclusion check with `exclusive()`, so the conflict is caught at `--check` time rather than silently at runtime. Before: ```java public static final Option<List<String>> KEY_REPLACE_FIELDS = Options.key("replace_fields").listType().noDefaultValue() .withFallbackKeys("replace_field"); // runtime sourceMap check, invisible to --check Map<String, Object> sourceMap = config.getSourceMap(); if (sourceMap.containsKey("replace_field") && sourceMap.containsKey("replace_fields")) { throw ...; } this.replaceFields.addAll(getRequiredOption(config, KEY_REPLACE_FIELDS)); ``` After: ```java @Deprecated public static final Option<List<String>> KEY_REPLACE_FIELD = Options.key("replace_field").listType().noDefaultValue(); public static final Option<List<String>> KEY_REPLACE_FIELDS = Options.key("replace_fields").listType().noDefaultValue(); // exclusive() validates at --check time: exactly one must be present OptionRule.builder() .exclusive(KEY_REPLACE_FIELD, KEY_REPLACE_FIELDS) .build(); if (config.getOptional(KEY_REPLACE_FIELDS).isPresent()) { this.fields = config.get(KEY_REPLACE_FIELDS); } else { this.fields = config.get(KEY_REPLACE_FIELD); } ``` ### D2. Exclusive with value constraints Use when exclusive options also need content validation (e.g. non-empty). > Framework support: > - [PR #11010](https://github.com/apache/seatunnel/pull/11010) — Map condition validators: `mapNotEmpty`, `mapContainsKey`, `mapContainsKeys` (merged) > - [PR #11022](https://github.com/apache/seatunnel/pull/11022) — Allow `exclusive`/`bundled` + `optional(condition)` coexistence (merged) Before: ```java // runtime imperative checks — invisible to --check if (config.get(SCHEMA) != null && config.get(TABLE_CONFIGS) != null) { throw new IllegalArgumentException("Cannot specify both 'schema' and 'table_configs'"); } if (config.get(SCHEMA) != null && config.get(SCHEMA).isEmpty()) { throw new IllegalArgumentException("'schema' must not be empty"); } ``` After: ```java import static org.apache.seatunnel.api.configuration.util.Conditions.mapNotEmpty; import static org.apache.seatunnel.api.configuration.util.Conditions.notEmpty; OptionRule.builder() .exclusive(SCHEMA, TABLE_CONFIGS) .optional(SCHEMA, mapNotEmpty(SCHEMA)) .optional(TABLE_CONFIGS, notEmpty(TABLE_CONFIGS)) .build(); ``` - `exclusive()` — exactly one must be present - `optional(option, condition)` — when present, value must satisfy the constraint - Both can coexist on the same option; constraints are only evaluated when the option is present ### D3. Custom structural validation (`ConditionExtension`) Use when built-in operators cannot express the validation — for example, validating internal structure of `List<Map<String, Object>>`, or enforcing constraints across nested child configs like `table_configs`. > Framework support: > > - [PR #11048](https://github.com/apache/seatunnel/pull/11048) — `ConditionOperator.EXTENSION` + `ConditionExtension<T>` interface (merged) Before: ```java // imperative structural check buried in buildWithConfig() — invisible to --check for (Map<String, Object> child : tableConfigs) { if (!child.containsKey("table_name")) { throw new IllegalArgumentException("each table config must contain 'table_name'"); } } // cross-element uniqueness check if (tableNames.size() != new HashSet<>(tableNames).size()) { throw new IllegalArgumentException("table names must be unique"); } ``` After: ```java import org.apache.seatunnel.api.configuration.util.ConditionExtension; import org.apache.seatunnel.api.configuration.util.Conditions; static class TableConfigsValidator implements ConditionExtension<List<Map<String, Object>>> { @Override public String description() { return "each entry must contain a non-empty 'table_name', " + "and all table names must be unique"; } @Override public boolean evaluate(ReadonlyConfig config, List<Map<String, Object>> value) throws OptionValidationException { if (value.isEmpty()) { return false; } Set<String> seen = new HashSet<>(); for (Map<String, Object> entry : value) { Object name = entry.get("table_name"); if (!(name instanceof String) || ((String) name).isEmpty()) { return false; } if (!seen.add((String) name)) { return false; } } return true; } } OptionRule.builder() .exclusive(TABLE_CONFIGS, SCHEMA) .optional(TABLE_CONFIGS, Conditions.extension(TABLE_CONFIGS, new TableConfigsValidator())) .build(); ``` * `ConditionExtension<T>` — implement `description()` (used in error messages and REST metadata) and `evaluate()` (validation logic, avoid I/O) * `Conditions.extension(Option<T>, ConditionExtension<T>)` — compile-time type binding between option and extension * Chains with `.and()` / `.or()` like any built-in operator * Return `false` for auto-composed error messages, or throw `OptionValidationException` for context-rich details ### E. Keep runtime checks for cases not suitable for declarative rules Do not migrate these into declarative rules: * external system/state validation (DB/metastore/network reachability) * complex parser/semantic validation (SQL parsing, advanced regex semantics) * runtime context checks (current time, execution topology state) ## Connectors Open for Claim | Type | Connector | Contributer | Status | PR | | --- | --- | --- | --- | --- | | Source | connector-cdc | @nzw921rx | Draft | #11023 | | Source | connector-edge-socket | @nzw921rx | Merged | #11384 | | Source | connector-fake | @Ayushkale11 | In review | #11032 | | Source | connector-google-sheets | | Todo | | | Source | connector-openmldb | | Todo | | | Source | connector-web3j | | Todo | | | Sink | connector-activemq | | Todo | | | Sink | connector-aerospike | | Todo | | | Sink | connector-assert | @cyl-uuu | In review | #11625 | | Sink | connector-bigquery | | Todo | | | Sink | connector-console | @asrajawat | Merged | #11477 | | Sink | connector-datahub | | Todo | | | Sink | connector-dingtalk | | Todo | | | Sink | connector-druid | | Todo | | | Sink | connector-email | @abolfazlmadanii | Merged | #11817 | | Sink | connector-fluss | | Todo | | | Sink | connector-google-firestore | | Todo | | | Sink | connector-hugegraph | | Todo | | | Sink | connector-hudi | @zhang-arvin | Claimed | | | Sink | connector-lance | | Todo | | | Sink | connector-mqtt | @ealeonraz | In review | #11461 | | Sink | connector-s3-redshift | | Todo | | | Sink | connector-selectdb-cloud | | Todo | | | Sink | connector-sensorsdata | | Todo | | | Sink | connector-sentry | | Todo | | | Sink | connector-slack | | Todo | | | Both | connector-amazondynamodb | @goutamadwant | In review | #11821 | | Both | connector-amazonsqs | | Todo | | | Both | connector-cassandra | | Todo | | | Both | connector-clickhouse | @Rajeshdevandla | In review | #11224 | | Both | connector-databend | | Todo | | | Both | connector-doris | @zhang-arvin | In review | #11858 | | Both | connector-easysearch | | Todo | | | Both | connector-elasticsearch | @nzw921rx | Merged | #11122 | | Both | connector-file | @RohanExploit | In review | #11881 | | Both | connector-graphql | | Todo | | | Both | connector-hbase | @goutamadwant | In review | #11803 | | Both | connector-hive | @zhang-arvin | Claimed | | | Both | connector-http | @junsoo22 | Claimed | | | Both | connector-iceberg | @ClaireLytt | Merged | #11675 | | Both | connector-influxdb | | Todo | | | Both | connector-iotdb | | Todo | | | Both | connector-iotdb-v2 | @liziing | In review | #11839 | | Both | connector-jdbc | @nzw921rx | Merged | #11106 | | Both | connector-kafka | @nzw921rx | Merged | #11157 | | Both | connector-kudu | | Todo | | | Both | connector-maxcompute | | Todo | | | Both | connector-milvus | @ZYZ666-RGB | Merged | #11504 | | Both | connector-mongodb | @AmanMishra1996 | In review | #11886 | | Both | connector-neo4j | | Todo | | | Both | connector-paimon | | Todo | | | Both | connector-prometheus | @goutamadwant | In review | #11738 | | Both | connector-pulsar | | Todo | | | Both | connector-qdrant | | Todo | | | Both | connector-rabbitmq | @cyl-uuu | In review | #11795 | | Both | connector-redis | @ss666 | Merged | #11195 | | Both | connector-rocketmq | @nzw921rx | Merged | #11158 | | Both | connector-sls | | Todo | | | Both | connector-socket | @nzw921rx | Merged | #11214 | | Both | connector-starrocks | @zhang-arvin | Claimed | | | Both | connector-tablestore | | Todo | | | Both | connector-tdengine | | Todo | | | Both | connector-typesense | | Todo | | ## Transforms Open for Claim | Type | Transform | Contributer | Status | PR | | --- | --- | --- | --- | --- | | Transform | CopyField | @nzw921rx | Merged | #11095 | | Transform | DataValidator | @nzw921rx | Merged | #11095 | | Transform | DynamicCompile | @nzw921rx | Merged | #11095 | | Transform | FieldEncrypt | @nzw921rx | Merged | #11095 | | Transform | FieldMapper | @nzw921rx | Merged | #11095 | | Transform | FilterField | @nzw921rx | Merged | #11095 | | Transform | FilterRowKind | @goutamadwant | In Review | #11763 | | Transform | JsonPath | @nzw921rx | Merged | #11095 | | Transform | Metadata | @nzw921rx | Merged | #11095 | | Transform | RegexExtract | @nzw921rx | Merged | #11095 | | Transform | Replace | @nzw921rx | Merged | #11095 | | Transform | RowKindExtractor| @nzw921rx | Merged | #11095 | | Transform | Split | @nzw921rx | Merged | #11095 | | Transform | SQL | @nzw921rx | Merged | #11095 | | Transform | TableFilter | @nzw921rx | Merged | #11095 | | Transform | TableMerge | @nzw921rx | Merged | #11095 | | Transform | DefineSinkType | @nzw921rx | Merged | #11095 | ## Note `connector-common` is a shared base module, not a standalone connector plugin, so it is excluded from claim rows. ## How to Contribute 1. **Pick a connector/transform**: Choose one from the lists above. 2. **Claim the task**: Comment on this issue (for example: `I would like to work on connector-kafka` or `I would like to work on FilterRowKind`). 3. **Implement**: - Migrate declarative-eligible validation rules from imperative `if/throw` to `optionRule()` + `Conditions.*`. - Keep runtime-only validation in runtime code paths (`*Config.java`, sink/source initialization) when it depends on external state or execution context. - Distinguish cross-field semantics explicitly: - required cross-field: `required(A, B, lessThanField(A, B))` - optional cross-field: `optional(A, B, lessOrEqualField(A, B))` 4. **Reference**: - Framework scope and audit baseline: [#10976](https://github.com/apache/seatunnel/issues/10976) - Declarative framework implementation: [#10977](https://github.com/apache/seatunnel/pull/10977) 5. **Submit PR**: Open a Pull Request against `dev`, and link it back to this issue. Thank you for your contribution. Please leave a message if you'd like to implement the declarative validation migration for any connector or transform. ## Code of Conduct I agree to follow this project's Code of Conduct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
