zhangshenghang opened a new issue, #13226:
URL: https://github.com/apache/seatunnel/issues/13226

   ## Motivation
   
   Today the `env` block of a SeaTunnel job config is consumed only at 
job/engine level (`job.mode`, `parallelism`, checkpoint settings, etc.). Plugin 
factories created through `TableFactoryContext` cannot see it, and SQL UDFs 
(`ZetaUDF` / `CalciteUdf`) have no way to read user-defined values from `env` 
at all.
   
   Users need to define job-scoped custom parameters once, for example:
   
   ```hocon
   env {
     job.mode = "BATCH"
     custom_parameters {
       udf_salt = "abc123"
       region = "cn-north"
     }
   }
   ```
   
   and read them from:
   
   - custom Source / Sink / Transform factories,
   - `ZetaUDF` implementations used by the `Sql` transform,
   - `CalciteUdf` implementations used by the `Calcite` transform.
   
   `EnvCommonOptions.CUSTOM_PARAMETERS` already exists and is already allowed 
by `EnvOptionRule`, but it is not exposed anywhere and is undocumented.
   
   ## Proposal
   
   ### 1. Expose env on the factory context (seatunnel-api)
   
   - Add `ReadonlyConfig getEnvOptions()` to `TableFactoryContext`.
   - Add env-aware constructor overloads to:
     - `TableSourceFactoryContext(options, classLoader, metadataConfig, 
envOptions)`
     - `TableSinkFactoryContext(catalogTable, options, classLoader, envOptions)`
     - `MultiTableFactoryContext(options, classLoader, sinks, envOptions)`
     - `TableTransformFactoryContext(catalogTables, options, classLoader, 
envOptions)`
   - Add `TableSinkFactoryContext.replacePlaceholderAndCreate(..., 
envOptions)`; env must stay outside placeholder rewriting.
   - Add trailing `envOptions` overloads to the five `FactoryUtil` entry points:
     `createAndPrepareSource`, `restoreAndPrepareSource`, 
`createAndPrepareSink`, `createMultiTableSink`, 
`createAndPrepareMultiTableTransform`.
   - All existing constructors/methods stay and delegate with an empty env, so 
existing connectors, tests and Flink/Spark starters are unaffected.
   
   ### 2. Wire the Zeta execution path
   
   `MultipleTableJobConfigParser` already has the parsed `envOptions` field; 
pass it to the source / restore-source / transform / multi-sink / sink factory 
calls. CLI and REST submission both go through this parser.
   
   Also pass env through `--dry-run=connect` (`DryRunConnectValidator`) for 
source, transform **and sink** validation so that dry-run semantics match 
runtime.
   
   ### 3. Expose env to SQL UDFs (seatunnel-transforms-v2)
   
   - `Sql` transform:
     - `SQLTransformFactory` reads `context.getEnvOptions()` and passes it to 
`SQLMultiCatalogFlatMapTransform` -> `SQLTransform`.
     - `SQLTransform` keeps env as a serializable field (transient SQL engine 
is rebuilt on the worker with the same env).
     - `SQLEngine.init(...)` gets a default 5-arg overload; `ZetaSQLEngine` 
stores env and calls `udf.open(envOptions)`.
     - `ZetaUDF` gets `default void open(ReadonlyConfig envOptions) throws 
Exception { open(); }`.
   - `Calcite` transform:
     - Same propagation through `CalciteTransformFactory` -> 
`CalciteMultiCatalogTransform` -> `CalciteTransform` -> `CalciteSQLEngine`.
     - `BuiltinFunctions.discoverAndRegister` calls `udf.open(envOptions)`.
     - `CalciteUdf` gets `default void open(ReadonlyConfig envOptions) { 
open(); }`.
     - `CalciteUdfContext.enter(tableId, rowKind, envOptions)` and 
`getEnvOptions()` expose env to static `eval` methods, because Calcite UDFs are 
invoked statically.
   
   Deep-copied env must be injected into the multi-catalog wrapper **before** 
inner per-table transforms are built: `AbstractMultiCatalogTransform` builds 
them in its own constructor, so env must live on the base class, not in 
subclass fields (subclass fields are still `null` at that point).
   
   ### 4. Documentation
   
   Document `env.custom_parameters` as the recommended namespace for plugin/UDF 
parameters, with examples for:
   
   - `TableFactoryContext#getEnvOptions()` in a connector factory,
   - `ZetaUDF#open(ReadonlyConfig)` / `evaluate`,
   - `CalciteUdf` via `CalciteUdfContext.current().getEnvOptions()`.
   
   Also document the limitations below.
   
   ## Design constraints
   
   - **Do not merge env into plugin options.** Env must remain a separate 
`getEnvOptions()` view. Merging would break unknown-key validation 
(`ConfigValidator.validateUnknownKeys`), sink table-placeholder rewriting, and 
connector `OptionRule` validation.
   - **Backward compatibility is a hard requirement.** No existing constructor, 
method signature, default value or config option may change.
   - Env is carried as `ReadonlyConfig` (already `Serializable`) so it follows 
transform serialization to workers.
   - `ZetaSQLEngine`'s existing 4-arg `init` must keep working and default to 
an empty env, because existing tests and callers invoke it directly.
   
   ## Non-goals (possible follow-ups)
   
   - Wiring Flink/Spark starters (their contexts keep receiving an empty env; 
behavior unchanged).
   - Extending `ConfigShadeUtils` decryption to the `env` block (env values are 
**not** decrypted today, so secrets should not be placed there).
   - Runtime row-level contexts (`SourceReader.Context`, writer context, etc.) 
exposing env.
   
   ## Known limitations to document
   
   - `ZetaUDF.resultType()` is invoked during schema inference before 
`open(env)`, so it cannot read env.
   - A Calcite UDF's static `eval` may also run during planning/constant 
folding, where `CalciteUdfContext.current()` is `null`; UDF code must 
null-check.
   - Calcite UDFs should not cache job-specific env in static fields (multiple 
jobs can share a JVM); use `CalciteUdfContext.current()` inside `eval`.
   - Adding a non-transient field to `SQLTransform` / `CalciteTransform` 
changes their implicit `serialVersionUID`, which may affect restore from 
checkpoints/savepoints written by older builds; this should be handled 
explicitly (declare a stable UID or document the incompatibility).
   - `env` is not decrypted by `ConfigShadeUtils` (it only processes 
`source`/`sink`/`transform`), so do not put encrypted values in `env`.
   
   ## Acceptance criteria
   
   1. Given a job with `env.custom_parameters.my_key = "my_value"`, a custom 
`ZetaUDF` receives that value through `open(ReadonlyConfig)` when the job runs 
on Zeta (local and cluster), including REST submission.
   2. A `CalciteUdf` can read the same value via 
`CalciteUdfContext.current().getEnvOptions()` inside `eval`.
   3. Source / Sink / Transform factories can read `context.getEnvOptions()` on 
the Zeta path.
   4. `--check-config` / `--dry-run=connect` do not report unknown option keys 
because of env, and dry-run passes env consistently to source, transform and 
sink validation.
   5. Flink/Spark execution and all existing connectors behave exactly as 
before (empty env).
   6. New unit tests cover: context getters + legacy empty default; 
`FactoryUtil` overloads; `Sql` UDF env end-to-end; `Calcite` UDF env 
end-to-end; parser-level env propagation.
   7. Documentation updated in `docs/en` and `docs/zh`.
   
   ## Additional context
   
   This capability depends on `EnvCommonOptions.CUSTOM_PARAMETERS` (already 
registered in `EnvOptionRule`), so no new config option is required. The change 
is additive API only; no existing signature is removed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to