DanielLeens commented on issue #12691: URL: https://github.com/apache/seatunnel/issues/12691#issuecomment-6059744533
Classification: F / Zeta state-storage feature design. @xinnyuli, thank you for the focused Azurite evidence. I reproduced the design reasoning against current `upstream/dev` rather than treating this as a generic Azure request. The checkpoint and IMap storage plugins already expose the same extension points that were used for merged GCS support in #12551: both configuration enums select an opt-in storage type, and IMap routes object stores through `CloudWriter` because `HdfsWriter` depends on append/flush visibility. Your WASB result therefore establishes a real IMap durability gap; it is not solved by the existing `storage.type: hdfs` workaround. Please scope the first contribution to Azure Blob / `wasb://` only, and add `storage.type: azure` to both checkpoint and IMap. Keeping checkpoint in the same opt-in type is useful for one validated configuration and coherent documentation, even though the existing HDFS-backed checkpoint write happens to work with WASB. Reuse the existing HDFS checkpoint implementation; for IMap, an `AzureWriter` following `GcsWriter`/`CloudWriter` is the required path so every WAL update is committed rather than relying on an open append stream. Do not include ADLS Gen2 / `abfs://` in this first PR. Its semantics need separate real-service evidence, and #12293 is a blocked, failing-build Connector-V2 ADLS PR; it does not implement or validate these Zeta storage paths. I found no active PR for Azure Blob checkpoint/IMap storage, so this issue can be the single implementation track. Before opening the PR, please post a short configuration contract covering the exact Blob URI/container key, supported authentication and the pass-through prefix for Hadoop Azure options. Keep any Azure jars provided/user-supplied as in #12551; do not introduce or shade a new dependency until its placement, license/NOTICE impact, and distribution loading are explicitly justified. The implementation needs deterministic tests for configuration validation and writer/reader selection, plus Azurite coverage for: checkpoint write/list/restore/delete; IMap write then a new-instance load; and an unclosed-writer/crash-equivalent recovery check. Add matching en/zh docs and state that IMap rewrites its current object on each update, so the performance trade-off is visible. Please keep the change additive: no existing `hdfs` behavior or default may change, no silent page-blob workaround, and no credentials in examples or logs. Once you have the contract and a minimal PR, link it here for review. No label or assignment changed in this pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
