Doris-Breakwater commented on issue #69645: URL: https://github.com/apache/doris/issues/69645#issuecomment-6418932228
Breakwater-GitHub-Analysis-Slot: slot_6d6fa6351941 **Triage: confirmed FE URI-handling defect at `8d2bd1e14b6`; recommend high priority because it can silently select a different existing object.** The report provides enough information to identify the key mutation. No repository code was changed. **Verified code path** - [`S3TableValuedFunction`](https://github.com/apache/doris/blob/8d2bd1e14b6d95d923d43e00857da326639cc466/fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java#L49-L58) normalizes the location through `StorageAdapter` before file discovery. The Azure adapter delegates to `StorageUriUtils.validateAndNormalizeAzureUri`. - In [`convertAzureToS3Style`](https://github.com/apache/doris/blob/8d2bd1e14b6d95d923d43e00857da326639cc466/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageUriUtils.java#L312-L335), HTTP(S) uses `URI.getPath()`, so `a%2Bb` becomes `a+b` in the intermediate `s3://c/a+b/1/a.csv`. The WASB/ABFS branches preserve the encoded text; S3-style input passes through. - [`AzureUri.decodeKey`](https://github.com/apache/doris/blob/8d2bd1e14b6d95d923d43e00857da326639cc466/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureUri.java#L158-L162) then applies `URLDecoder`, turning that literal `+` into a space. It also changes an originally unencoded `+` into a space across the supported schemes. - [`ExternalFileTableValuedFunction.parseFile`](https://github.com/apache/doris/blob/8d2bd1e14b6d95d923d43e00857da326639cc466/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java#L166-L208) uses glob discovery for exact paths too. [`AzureFileSystem.globListWithLimit`](https://github.com/apache/doris/blob/8d2bd1e14b6d95d923d43e00857da326639cc466/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystem.java#L393-L451) builds its matcher and listing prefix from the already changed key. `AzureObjStorage.listObjects` passes the parsed prefix to the Azure SDK. This explains how the space-named object can match without an error, for both exact and glob paths. **Independent mechanism check** An isolated JDK 17 probe compiled the unchanged `AzureUri` from this commit and the verbatim `convertAzureToS3Style` method body, with a minimal exception stand-in. All 14 asserted observations matched the current behavior, including: | Input path form | Key after normalization and parsing | | --- | --- | | HTTPS `a%2Bb/1/a.csv` | `a b/1/a.csv` | | HTTPS `a%2Bb/*/a.csv` | `a b/*/a.csv` | | ABFSS or S3-style `a%2Bb/1/a.csv` | `a+b/1/a.csv` | | HTTPS `a%20b/1/a.csv` | `a b/1/a.csv` | | HTTPS `a%252Bb/1/a.csv` | `a+b/1/a.csv` | | S3-style `a%252Bb/1/a.csv` | `a%2Bb/1/a.csv` | | ABFSS literal `a+b/1/a.csv` | `a b/1/a.csv` | The `%252B` result needs clarification in the report: under the stated decode-once contract, it should identify a name containing literal `%2B`, not `+`. Apparent success for a plus-named object would still reflect double decoding. The same probe showed HTTPS `%3F`/`%23` becoming delimiters that truncate the key during parsing; include those as regression cases. These are verified parser results and source-based tracing, **not a live Azure/Doris query reproduction**. The mutation occurs before credential-dependent SDK access, supporting the reported authentication independence; SharedKey/SAS/OAuth2 query results were not independently exercised. **Suggested next steps** 1. Preserve the encoded HTTP(S) path with `getRawPath()` during normalization, and make `decodeKey` decode percent escapes once while preserving literal `+`. Both changes are needed to address the reported case and the related literal-plus case. The proposed `raw.replace("+", "%2B")` before `URLDecoder.decode` is consistent with that contract. 2. Add a regression that traverses **normalization followed by Azure parsing/listing**, for exact and glob paths across HTTP(S), WASB(S), ABFS(S), and Azure S3-style input. Existing `AzureUriTest` and `globListWithLimit_preservesBaseForEncodedInputKeys` test encoded input directly and do not exercise the preceding HTTPS rewrite. Cover `%2B`/`%2b`, literal `+`, `%20`, `%252B`, encoded `?`/`#`, and a `+08:00` partition value. 3. Validate against distinct plus-, space-, and literal-`%2B`-named objects, checking both selected paths and returned sentinel values. Then run the credential variants as integration coverage and assess affected release branches from their actual code; the first affected release is not established here. **Remaining evidence:** no additional logs or profile are needed to establish this parser defect. Live acceptance testing and the exact intended object/result for the `%252B` control remain outstanding. If integration results differ, provide a sanitized query, FE/BE build hashes, selected file paths, and relevant FE/BE logs; remove account keys, SAS tokens, and OAuth secrets. For the specific encoded-plus case, the reporter's working `s3://c/a%2Bb/1/a.csv` or encoded ABFSS form is a reasonable temporary workaround. Avoid literal `+` and treating double encoding as a general workaround. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
