zhangshenghang commented on issue #13226:
URL: https://github.com/apache/seatunnel/issues/13226#issuecomment-6797291462

   Thanks for the review. Agreed on the staged delivery and on keeping `env` 
out of the credential path. I have documented the compatibility contract below 
and will keep stage 1 and stage 2 reviewable separately.
   
   ## Stage 1 — factory env view + Zeta wiring (PR #13233)
   
   Implemented in the first PR:
   
   - Immutable `ReadonlyConfig getEnvOptions()` on `TableFactoryContext`; empty 
env is the default and is never merged into plugin options.
   - Env-aware constructors for `TableSourceFactoryContext`, 
`TableSinkFactoryContext`, `MultiTableFactoryContext` and 
`TableTransformFactoryContext`.
   - Env-aware `FactoryUtil` overloads for `createAndPrepareSource`, 
`restoreAndPrepareSource`, `createAndPrepareSink`, `createMultiTableSink` and 
`createAndPrepareMultiTableTransform`.
   - `MultipleTableJobConfigParser` passes env through the Zeta source, 
restore-source, transform, multi-sink and sink paths (CLI and REST share this 
parser).
   - `DryRunConnectValidator` passes env to source, transform and sink dry-run 
contexts.
   - Flink/Spark behavior is unchanged (empty env).
   - Focused API tests, a Zeta-path parser test and EN/ZH documentation.
   
   `env.custom_parameters` is explicitly **not** a secret store: 
`ConfigShadeUtils` only decrypts `source`/`sink`/`transform`, and the docs 
state this.
   
   ## Constructor / overload compatibility table
   
   All rows are additive: no existing signature is removed or changed.
   
   | API | Existing signature (preserved) | New overload |
   | --- | --- | --- |
   | `TableFactoryContext` | `(ReadonlyConfig, ClassLoader)` | 
`(ReadonlyConfig, ClassLoader, ReadonlyConfig envOptions)` |
   | `TableSourceFactoryContext` | `(options, classLoader)`, `(options, 
classLoader, metadataConfig)` | `(options, classLoader, metadataConfig, 
envOptions)` |
   | `TableSinkFactoryContext` | `(catalogTable, options, classLoader)` | 
`(catalogTable, options, classLoader, envOptions)` |
   | `TableSinkFactoryContext.replacePlaceholderAndCreate` | `(catalogTable, 
options, classLoader, excludeKeys)` | `(catalogTable, options, classLoader, 
excludeKeys, envOptions)` |
   | `MultiTableFactoryContext` | `(options, classLoader, sinks)` | `(options, 
classLoader, sinks, envOptions)` |
   | `TableTransformFactoryContext` | `(catalogTables, options, classLoader)` | 
`(catalogTables, options, classLoader, envOptions)` |
   | `FactoryUtil.createAndPrepareSource` | 6-arg form | 7-arg form with 
trailing `envOptions` |
   | `FactoryUtil.restoreAndPrepareSource` | 7-arg form | 8-arg form with 
trailing `envOptions` |
   | `FactoryUtil.createAndPrepareSink` | 6-arg form | 7-arg form with trailing 
`envOptions` |
   | `FactoryUtil.createMultiTableSink` | 3-arg form | 4-arg form with trailing 
`envOptions` |
   | `FactoryUtil.createAndPrepareMultiTableTransform` | 4-arg form | 5-arg 
form with trailing `envOptions` |
   | `SQLEngine.init` | 4-arg abstract method | 5-arg default method delegating 
to the 4-arg form |
   | `ZetaUDF.open` | `default void open() throws Exception` | `default void 
open(ReadonlyConfig) throws Exception` delegating to `open()` |
   | `CalciteUdf.open` | `default void open()` | `default void 
open(ReadonlyConfig)` delegating to `open()` |
   | `CalciteUdfContext.enter` | `(tableId, rowKind)` | `(tableId, rowKind, 
envOptions)` |
   
   Old constructors and methods delegate with an empty environment. Adding 
constructors, overloads and default interface methods is binary compatible; 
existing third-party factories and UDF implementations keep their current 
behavior.
   
   ## Serialization compatibility treatment (stage 2)
   
   - `ReadonlyConfig` is already `Serializable` (`serialVersionUID = 1L`) and 
env values from HOCON rendering are Jackson map/list/scalar types.
   - `SQLTransform` and `CalciteTransform` carry env as a non-transient field; 
their SQL engines are `transient` and are rebuilt with the same env on the 
worker.
   - A legacy stream does not contain the new `envOptions` field. 
Deserialization leaves it `null`; `open()` treats `null` as an empty env, and 
`AbstractMultiCatalogTransform#getEnvOptions()` is null-safe. Legacy transforms 
therefore keep the previous (empty-env) behavior.
   - To keep old streams readable, the pre-change implicit UIDs were computed 
from `dev` with `serialver` and pinned explicitly:
     - `SQLTransform`: `5502594799389514109L`
     - `CalciteTransform`: `426108317364037266L`
     - `AbstractMultiCatalogTransform` already declares an explicit UID and is 
unaffected.
   - New streams remain readable by older builds (unknown fields are ignored).
   
   ## Stage 2 — SQL / Calcite UDF integration (follow-up PR)
   
   Planned/covered tests:
   
   - Local submission (Zeta container E2E) for both `Sql` and `Calcite`.
   - REST submission through the in-process Zeta cluster, asserting the job 
reaches `FINISHED` with an Assert sink (the job fails if env does not reach the 
UDF).
   - Job isolation when UDF classes share a JVM: two transforms/engines with 
different env values, executed interleaved; Zeta keeps env per UDF instance, 
Calcite reads it from the per-row context.
   - `CalciteUdfContext` cleanup: the thread-local scope is removed after 
`execute`, so `current()` returns `null` and no job-specific env remains.
   - Restore/serialization: round-trip of the parsed transform plus the 
UID/legacy-stream treatment above.
   
   I will narrow PR #13233 to stage 1 and keep the UDF work on a stacked branch 
for the follow-up PR once the factory API contract is settled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to