xinnyuli commented on issue #12691: URL: https://github.com/apache/seatunnel/issues/12691#issuecomment-6081733531
@DanielLeens Thanks, scoping to Blob / `wasb://` with `storage.type: azure` for both checkpoint and IMap. Here is the configuration contract I plan to implement, mirroring the GCS one: **Type and URI** - `storage.type: azure` (new, opt-in; `hdfs` and all other types unchanged). - `azure.container` (required): the container URI, `wasbs://<container>@<account>.blob.core.windows.net` (TLS, recommended) or `wasb://...`. It becomes `fs.defaultFS`, and `namespace` stays the path inside the container as for the other types. - Validation fails fast with a clear message if the key is missing, if the scheme is not `wasb://`/`wasbs://` (`abfs://`/`abfss://` are rejected and pointed to as out of scope), or if the URI has no `<container>@<account>` authority. - The container must already exist. hadoop-azure does not create it on first write, which I confirmed with Azurite. The docs will say so. **Hadoop settings set by the plugin** - `fs.wasb.impl = org.apache.hadoop.fs.azure.NativeAzureFileSystem`, `fs.wasbs.impl = org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure`. - Checkpoint honours `disable.cache` like the other checkpoint types; IMap leaves the FileSystem cache at the Hadoop default, like IMap S3/OSS/GCS. **Authentication** (passed through, no new option names) - Every `fs.azure.*` key is passed to hadoop-azure unchanged, the same as `fs.gs.*` for GCS. - Supported and documented: storage account key, `fs.azure.account.key.<account>.blob.core.windows.net`, and container SAS token, `fs.azure.sas.<container>.<account>.blob.core.windows.net`. - Not supported in this PR: managed identity / OAuth (WASB does not support them; that belongs with ABFS) and anonymous access. - Examples use placeholders only, and the plugin will not log configuration values. **IMap writer** - `AzureWriter extends CloudWriter` (identifier `azure`), registered next to `GcsWriter`. Every WAL update rewrites the current object, rolling over at `block.size` (default 1 MB). The docs will state this trade-off. **Dependencies** - hadoop-azure 3.1.4 (matching the shipped Hadoop) and its Azure storage SDK stay user-supplied in `lib/`, as with GCS. In the poms they appear only with `test` scope for the Azurite tests; nothing is bundled or shaded. The PR will list the exact jars after checking them against a clean `lib/`. **Tests** - Unit: configuration validation (missing key, wrong scheme, `wasb`/`wasbs` accepted, `fs.azure.*` pass-through, impl keys set) and `azure` writer selection. - Azurite: checkpoint write/list/restore/delete; IMap write, then load from a new instance; IMap unclosed-writer (crash-equivalent) recovery. One detail I'll handle in the PR: hadoop-azure's emulator mode is fixed to `127.0.0.1:10000`, so the Azurite container needs that host port. Does this look right before I open the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
