mmadzia-sfsh opened a new issue, #67544: URL: https://github.com/apache/doris/issues/67544
> **Note on authorship:** this report was researched and written by **Claude Code** (Anthropic's CLI > coding agent), driven by and reviewed by the account owner. The source analysis, the live > reproduction, and the cleanup were all executed as described below against a real Doris 4.1.3-rc02 > cluster on a disposable table. Paths, account names and hostnames in the repro have been > genericised; everything else is verbatim. ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. Searched for `convertToS3Style`, `Failed to create LocationPath`, `delete_pos`, `LocationPath`, and `iceberg delete file abfss s3` — no existing report. ### Version - Reproduced on **`doris-4.1.3-rc02-7126cf65d96`** (3 FEs, `SHOW FRONTENDS`). - Also present on **`master` @ `b58b2c53ff56354c692c2dc7634d724d8780838f`** by code inspection (see *Anything Else?* — the relevant BE writer is byte-identical and the FE lost the field on the delete lane). - Catalog: Iceberg REST (Lakekeeper 0.13.1), storage **ADLS Gen2 (`abfss://`)**, Iceberg format-version 2, `write.delete.mode=merge-on-read`. ### What's Wrong? On an Iceberg catalog backed by **ADLS Gen2**, a SQL `UPDATE` or `DELETE` commits a snapshot whose **position-delete file is recorded in the manifest with Doris's internal `s3://`-normalized path** instead of the real `abfss://` URI. Every subsequent read of the table then fails permanently: ``` ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to create LocationPath for location: s3://<container>/<uuid>/data/delete_pos_<uuid>_125613.zstd.parquet ``` `INSERT` on the same table is unaffected — its data files are written with correct `abfss://` paths. **The same `UPDATE` statement writes a correct `abfss://` data file and a broken `s3://` delete file**, which isolates the defect to the position-delete lane. The table is not recoverable through SQL — the bad path is durable Iceberg metadata. It has to be rolled back to the previous snapshot out-of-band (we use pyiceberg `manage_snapshots().rollback_to_snapshot(...)`, followed by `REFRESH TABLE`). **Root cause.** `abfss://<container>@<account>.dfs.core.windows.net/<key>` is deliberately normalized to `s3://<container>/<key>` for the BE (the account is re-attached BE-side from `azure.account_name`). That internal form is supposed to stay internal: the FE sends the BE **both** the normalized path (`output_path`) and the original (`original_output_path`), and the writer is expected to report the *original* back for the manifest. The **data** writer does this. The **position-delete** writer does not — it reports its normalized `output_path`, and that string is written straight into the manifest. On the read side, `LocationPath.findStorageProperties()` then sees scheme `s3` on an Azure-only catalog. `AzureProperties.getStorageName()` is `"AZURE"`, and the legacy compatibility fallback only matches a storage config literally named `"s3"` — so nothing matches and construction throws. Note this is *not* a catalog misconfiguration and *not* the REST catalog's doing: the catalog returns correct fully-qualified `abfss://` locations (verified on the wire), and the table has no `write.data.path` / `write.object-storage.path` / `write.folder-storage.path` set. The `s3://` string does not exist anywhere until Doris synthesises it. ### What You Expected? `UPDATE` / `DELETE` on an `abfss://`-backed Iceberg table should record position-delete files in the manifest using the same original `abfss://` URI scheme that data files already use, so the table remains readable after the commit. Failing that, Doris should reject the DML rather than commit a snapshot that permanently breaks reads. ### How to Reproduce? Iceberg REST catalog on ADLS Gen2. Catalog roughly: ```sql CREATE CATALOG `iceberg_adls` PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "uri" = "http://<rest-catalog-host>:8181/catalog", "warehouse" = "<warehouse>", "fs.azure.support" = "true", "azure.account_name" = "<account>", "azure.account_key" = "<key>" ); ``` Then, on a throwaway table: ```sql -- 1. create CREATE TABLE iceberg_adls.<ns>.zz_probe ( id INT NOT NULL, val STRING NULL ) ENGINE = iceberg PROPERTIES ( 'write-format' = 'parquet', 'compression-codec' = 'zstd', 'format-version' = '2', 'write.delete.mode' = 'merge-on-read' ); -- 2. insert INSERT INTO iceberg_adls.<ns>.zz_probe VALUES (1,'a'),(2,'b'),(3,'c'); -- 3. reads fine, and every file is abfss:// SELECT * FROM iceberg_adls.<ns>.zz_probe ORDER BY id; SELECT content, file_path FROM iceberg_adls.<ns>.zz_probe$files; -- content | file_path -- 0 | abfss://<container>@<account>.dfs.core.windows.net/<uuid>/data/....zstd.parquet -- 4. one row-level update UPDATE iceberg_adls.<ns>.zz_probe SET val='UPDATED' WHERE id = 2; -- 5. the delete file is s3://, the data files (including the one this UPDATE just wrote) are abfss:// SELECT content, file_path FROM iceberg_adls.<ns>.zz_probe$files; ``` Observed at step 5: | content | file_path | |---|---| | 0 (data) | `abfss://<container>@<account>.dfs.core.windows.net/<uuid>/data/….zstd.parquet` | | 0 (data — **written by the UPDATE**) | `abfss://<container>@<account>.dfs.core.windows.net/<uuid>/data/….zstd.parquet` | | 1 (position delete) | **`s3://<container>/<uuid>/data/delete_pos_<uuid>_125613.zstd.parquet`** | ```sql -- 6. the table is now permanently unreadable SELECT * FROM iceberg_adls.<ns>.zz_probe ORDER BY id; -- ERROR 1105 (HY000): errCode = 2, detailMessage = -- Failed to create LocationPath for location: s3://<container>/<uuid>/data/delete_pos_<uuid>_125613.zstd.parquet ``` `EXPLAIN` still succeeds at step 6 — planning does not resolve the delete file, so only execution fails. We ran this twice on two independent fresh tables; identical result both times. Both were dropped afterwards. Cluster: 3 FE / 3 BE, `doris-4.1.3-rc02-7126cf65d96`, Iceberg REST catalog (Lakekeeper 0.13.1) over ADLS Gen2 with shared-key auth. ### Anything Else? **Where it goes wrong, with line references.** *Normalization (by design).* `AzurePropertyUtils.validateAndNormalizeUri()` — javadoc literally reads *"@return a normalized `s3://`-style URI"* — calls `convertToS3Style()`, which maps `abfss://<container>@<account>.dfs.core.windows.net/<key>` → `s3://<container>/<key>`, dropping the account. Unconditional; the only bypass is `isOneLakeLocation()` (MS Fabric `*.dfs.fabric.microsoft.com`, returned unchanged). There is no user-facing switch to disable it. (`fe/fe-core/.../datasource/property/storage/AzurePropertyUtils.java` on `branch-4.1`; FE storage properties were refactored into `fe/fe-filesystem/fe-filesystem-azure/` on master.) *The FE hands the BE both forms.* On `branch-4.1`, `fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java:241-243`: ```java String originalLocation = writeSchemaContext .map(IcebergWriteSchemaContext::getDataLocation) .orElseGet(() -> IcebergUtils.dataLocation(icebergTable)); LocationPath locationPath = LocationPath.of(originalLocation, storagePropertiesMap); tSink.setOutputPath(locationPath.toStorageLocation().toString()); // s3://... tSink.setOriginalOutputPath(originalLocation); // abfss://... ``` *Data writer uses the original — correct.* `be/src/exec/sink/writer/iceberg/viceberg_table_writer.cpp:618-646` builds `original_write_path` from `iceberg_table_sink.original_output_path` and passes it in `WriteInfo`. *Position-delete writer does not.* `be/src/exec/sink/writer/iceberg/viceberg_delete_file_writer.cpp` contains **zero** references to `original_output_path` / `original_write_path` on both `branch-4.1` and `master`: ```cpp // line 50 _output_path(output_path), // line 76 io::FileDescription file_description = {.path = _output_path, .fs_name {}}; // line 150 <-- the normalized s3:// path is what gets reported for the manifest commit_data.__set_file_path(_output_path); ``` *And the FE writes that string verbatim.* `IcebergWriterHelper.convertToDeleteFiles()` — `String deleteFilePath = commitData.getFilePath(); … .withPath(deleteFilePath)` (line ~328 on `branch-4.1`, ~326 on master). By contrast the data path at line ~95/133 receives an already-corrected value from the BE. **On master the delete lane is missing the plumbing at the FE too.** In `fe/fe-connector/fe-connector-iceberg/.../IcebergWritePlanProvider.java` (@ `b58b2c53`): | method | line | sets `originalOutputPath`? | |---|---|---| | `buildRewriteSink` | 747-752 | ✅ yes | | **`buildDeleteSink`** | **816-817** | ❌ **no** — sets `outputPath` + `tableLocation` only | | `buildMergeSink` | 889-891 | ✅ yes | **Suggested fix.** Thread `original_output_path` through the delete lane exactly as the data lane already does — set it in `buildDeleteSink` on the FE, and have `VIcebergDeleteFileWriter` keep the original alongside `_output_path` and report *that* in `commit_data.__set_file_path(...)`, while still writing the file through the normalized path. Any storage whose URI is rewritten on the way to the BE is affected by the same asymmetry, so this is not Azure-specific in principle — Azure is just where the rewrite is unconditional. **Not a workaround:** `ALTER TABLE … SET ('write.data.path'='abfss://…')` is rejected (`Unknown table property: [write.data.path]`), and setting it out-of-band would not help — it only changes which directory is fed into the normalizer, and the delete writer still reports the post-normalization value. Adding S3 credentials to the catalog only makes `s3://<container>/…` resolve to a nonexistent S3 bucket. Related but **not** a fix for this: #64028 / #64042 (`IcebergUtils.dataLocation` priority chain) — that only selects *which* directory; all branches still pass through `convertToS3Style()`. We are on a build that contains it and still reproduce. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! Happy to test a patch against the reproduction above and report back. ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
