zhangshenghang opened a new pull request, #13233:
URL: https://github.com/apache/seatunnel/pull/13233
## Contribution acknowledgement
Yes — I am glad to contribute this feature to **Apache SeaTunnel**.
## Summary
This PR exposes the job-level `env` block to plugin factories and SQL UDFs,
so users can define job-scoped values once (for example under
`env.custom_parameters`) and read them from:
- Source / Sink / Transform factories through
`TableFactoryContext#getEnvOptions()`.
- `ZetaUDF#open(ReadonlyConfig)` in the `Sql` transform.
- `CalciteUdfContext#current().getEnvOptions()` inside a static `eval`
method in the `Calcite` transform.
The `env` options are intentionally **not merged into plugin options**, so
existing option-rule validation, unknown-key checks and sink table-placeholder
rewriting are unaffected.
Closes #13226.
## Motivation
Today `env` is only consumed at job/engine level. Plugin factories cannot
see it and SQL UDFs have no way to read user-defined parameters from it.
`EnvCommonOptions.CUSTOM_PARAMETERS` already exists and is allowed by
`EnvOptionRule`, but it was neither exposed nor documented.
Example:
```hocon
env {
job.mode = "BATCH"
custom_parameters {
my_key = "my_value"
}
}
```
## Changes
### seatunnel-api
- Add `TableFactoryContext#getEnvOptions()` and an empty-env default.
- Add env-aware constructors to `TableSourceFactoryContext`,
`TableSinkFactoryContext`, `MultiTableFactoryContext` and
`TableTransformFactoryContext`.
- Add env-aware overloads to all five `FactoryUtil` entry points
(`createAndPrepareSource`, `restoreAndPrepareSource`, `createAndPrepareSink`,
`createMultiTableSink`, `createAndPrepareMultiTableTransform`).
- Preserve all existing constructors and methods; they delegate with an
empty env.
### Zeta execution path
- `MultipleTableJobConfigParser` passes its parsed `envOptions` to source,
restore-source, transform, multi-table-sink and sink creation.
- `DryRunConnectValidator` receives env and passes it to source, transform
and sink dry-run contexts.
- CLI and REST submissions share the same parser, so both get the capability.
### SQL transforms and UDFs
- `Sql`: the factory passes env to `SQLMultiCatalogFlatMapTransform` ->
`SQLTransform`; the transform keeps env as a serializable field and injects it
into the transient SQL engine on the worker.
- `SQLEngine#init(..., envOptions)` gets a default overload; `ZetaSQLEngine`
stores env and calls `ZetaUDF#open(ReadonlyConfig)`.
- The multi-table wrapper stores env on the base class **before** building
per-table transforms, because `buildTransform` is invoked from the base
constructor (subclass fields are still `null` at that point).
- `Calcite`: env flows through `CalciteTransformFactory` ->
`CalciteMultiCatalogTransform` -> `CalciteTransform` -> `CalciteSQLEngine`;
`BuiltinFunctions` opens UDFs with env; `CalciteUdfContext` exposes env to
static `eval` calls.
### Documentation
- Documented `env.custom_parameters` and the plugin/UDF access patterns in
`docs/en` and `docs/zh` (`concepts/config.md` and
`configuration/JobEnvConfig.md`), including limitations.
## Compatibility
- All existing constructors and method signatures are preserved as overloads
delegating with an empty env.
- Adding default methods to `ZetaUDF`, `CalciteUdf` and `SQLEngine` is
binary compatible for existing implementations.
- Flink/Spark execution is unchanged: their factory contexts and UDFs
receive an empty env.
- `env` is not decrypted by `ConfigShadeUtils` (which only processes
`source`/`sink`/`transform`), so encrypted/sensitive values should not be
placed in `env`; this is documented.
- Adding a field to `SQLTransform` / `CalciteTransform` changes their
implicit `serialVersionUID`; checkpoint/savepoint restore from older builds is
not guaranteed across this change.
## Tests
### Unit / integration tests (all passing)
```bash
./mvnw -pl seatunnel-api test \
-Dtest='TableFactoryContextEnvTest,FactoryUtilEnvOptionsTest'
./mvnw -pl seatunnel-transforms-v2 test \
-Dtest='SQLTransformFactoryTest,CalciteTransformFactoryTest,SQLMultiCatalogFlatMapTransformTest,AbstractMultiCatalogTransformTest,ZetaSQLEngineTest,CalciteSQLEngineTest,ZetaUdfEnvOptionsTest,CalciteUdfEnvOptionsTest'
./mvnw -pl seatunnel-engine/seatunnel-engine-client test \
-Dtest=MultipleTableJobConfigParserTest
./mvnw -pl seatunnel-core/seatunnel-starter test \
-Dtest=SeaTunnelConfValidateCommandTest
```
Results: 8 API tests, 387 transform tests, 17 parser tests and 30 starter
tests passed. The parser tests also serialize/deserialize the parsed transform
to simulate the client -> worker hop.
### E2E tests (passing on Zeta, Testcontainers)
```bash
./mvnw -pl
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf
verify \
-DskipUT=true -DskipIT=false \
-Dit.test=ExampleUdfIT#testCustomUdfReadsEnvOptions
./mvnw -pl
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2
verify \
-DskipUT=true -DskipIT=false \
-Dit.test=TestCalciteIT#testCalciteUdfReadsEnvOptions
```
- `custom_udf_env.conf`: `env.custom_parameters.probe` reaches
`ZetaUDF#open(ReadonlyConfig)` and the Assert sink verifies `env-value:Hello
World`.
- `calcite_env_udf.conf`: the same value is read through
`CalciteUdfContext#current().getEnvOptions()` in a static `eval`; the test is
disabled on Flink/Spark because env propagation is Zeta-only.
### Formatting / packaging
```bash
./mvnw spotless:apply
./mvnw -pl <changed modules> -DskipTests verify
```
## Known limitations (documented)
- Flink/Spark provide an empty env to factories and UDFs in this change.
- `ZetaUDF.resultType(...)` runs before `open(...)`, so it cannot read env
yet.
- A Calcite UDF static `eval` may run during planning constant folding where
`CalciteUdfContext.current()` is `null`; UDFs must null-check.
- Calcite UDFs should not store job-scoped env in static fields; read it
from the per-row context instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]