924060929 commented on code in PR #66348:
URL: https://github.com/apache/doris/pull/66348#discussion_r3701905586
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureObjStorage.java:
##########
@@ -524,4 +495,15 @@ private static String toBlockId(int partNum) {
byte[] bytes =
ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(partNum).array();
return Base64.getEncoder().encodeToString(bytes);
}
+
+ static String multipartBlockId(String uploadId, int partNum) {
+ int uploadNamespace = 0x811C9DC5;
+ for (byte value : uploadId.getBytes(StandardCharsets.UTF_8)) {
+ uploadNamespace = (uploadNamespace ^ (value & 0xFF)) * 0x01000193;
+ }
+ int namespacedPart = uploadNamespace + partNum;
Review Comment:
[P1] Do not compress writer identity into a colliding 32-bit block namespace
The four-byte length fixes compatibility with legacy residual blocks, but
`FNV-1a-32(uploadId) + partNum` cannot isolate concurrent writers. For example,
these two valid UUIDs both hash to `541563024` under this exact implementation:
- `c8e3f981-50d5-46d3-8c3a-2d2723f686ee`
- `dd3cf642-9f55-4b6b-957d-0616633116ea`
For part 1 both therefore generate `kZhHIA==`. If writer A stages that ID,
writer B stages the same ID, and A commits its block list, Azure commits B’s
last-staged bytes while A reports success. Partial overlap is also possible
because each upload occupies the additive range `[hash + 1, hash + partCount]`.
This can publish a mixed/corrupt data file whose Iceberg metadata belongs to
another writer.
The current test only proves `upload-a` and `upload-b` differ; it cannot
establish uniqueness in a 32-bit space. Azure documents that the last upload of
a repeated uncommitted block ID is what a subsequent block-list commit selects:
https://learn.microsoft.com/en-us/rest/api/storageservices/put-block
Please use target-level coordination such as a lease, or another publication
design that preserves both legacy-length compatibility and deterministic writer
isolation. Changing the 32-bit hash function cannot fix the namespace-size
limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
