xylaaaaa opened a new pull request, #69392:
URL: https://github.com/apache/doris/pull/69392
### What problem does this PR solve?
Issue Number: None
Related PR: #69082
Problem Summary: DELETE, UPDATE and MERGE on an Iceberg table whose location
FE normalizes for BE
(oss://, cos://, cosn://, obs://, bos://, gs://, s3a://, s3n://, and Azure
abfs/abfss/wasb/wasbs
except OneLake) record the new delete files in Iceberg metadata with the
internal s3:// address
instead of the table's own scheme. Both v2 position delete files
(delete_pos_*.parquet) and v3
deletion vector files (delete_dv_*.puffin) are affected; data files are not.
Reproduce on an Iceberg table located at `oss://bucket/...`:
```sql
CREATE TABLE t (id INT, v STRING) PROPERTIES ('format-version' = '2');
INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
DELETE FROM t WHERE id = 1;
SELECT file_path FROM t$delete_files; --
s3://bucket/.../data/delete_pos_xxx.parquet
SELECT file_path FROM t$data_files; -- oss://bucket/.../data/...
```
Doris itself still reads the table, because it normalizes every path it
reads anyway, but other
engines resolve the file system by scheme and fail: Spark or Trino reading
the table, or running
expire_snapshots / remove_orphan_files, look the delete file up as s3 (e.g.
OSSFileIO rejects the
s3 scheme, HadoopFileIO usually has no file system for it, ResolvingFileIO
goes to AWS S3,
ADLSFileIO rejects it as an invalid ADLS URI).
Cause: data files are written under write_path (the s3:// form BE can
address) and committed
under original_write_path (VIcebergPartitionWriter). VIcebergDeleteSink had
no such split: it
generates delete file paths under output_path and reports exactly that path
for the commit
(VIcebergDeleteFileWriter sets file_path to its output path, and the puffin
path is reported
as written).
Fix: apply the same split to delete files. VIcebergDeleteSink still writes
under output_path but
commits under table_location, the raw data location FE already sends, e.g.
written to
s3://b/db/t/data/delete_x.parquet and committed as
oss://b/db/t/data/delete_x.parquet. Every FE
version sets table_location to the raw form of output_path's directory
(delete sink and merge
sink alike), so no protocol change is needed. That contract is now written
on table_location in
DataSinks.thrift (comments only) and asserted by an FE unit test, since
breaking it would record
delete files that do not exist without any error.
Delete files that an older version already committed with s3:// keep that
address until they
are rewritten.
### Release note
Fix Iceberg DELETE/UPDATE/MERGE recording delete files with an s3:// path
instead of the table's
own location (e.g. oss:// or abfss://), which made other engines fail to
read the table.
### Check List (For Author)
- Test: Unit Test (VIcebergDeleteSinkTest.TestCommitPathUsesTableLocation,
IcebergWritePlanProviderTest.planWriteDeleteAndMergeSinksKeepTableLocationInTheOutputDirectory;
not run locally, CI runs them)
- Behavior changed: Yes. Delete files are committed under table_location.
- Does this need documentation: No
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
