[
https://issues.apache.org/jira/browse/HADOOP-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126163#comment-18126163
]
ASF GitHub Bot commented on HADOOP-19973:
-----------------------------------------
vt921102 commented on PR #8692:
URL: https://github.com/apache/hadoop/pull/8692#issuecomment-6116784601
Thanks @anujmodi2021 — the `branch-3.5` validations are complete.
I ran the full `hadoop-azure` integration suite **on Windows**, which
matters for
this particular change: the defect only reproduces where `File.separator` is
a
backslash, so the earlier trunk run (macOS) could not exercise the failing
platform. This is the first integration validation of the fix on the OS
where the
bug actually occurs.
### Environment
- Windows 11 Pro 24H2, 4 vCPU / 8 GiB
- JDK 17 (Microsoft Build of OpenJDK 17.0.20.1), Maven via the project
`mvnw.cmd`
- Visual Studio 2022 Build Tools, MSVC v143, Windows SDK 10.0.22621
- ADLS Gen2 account in Azure **Sweden Central**, **HNS enabled**,
auth = **OAuth 2.0 client credentials** (`ClientCredsTokenProvider`)
The storage account belongs to my employer, so I have not named it here. I
am happy
to share the endpoint with a committer privately if the project requires the
specific account name on record.
### Commits tested
| Role | Commit |
|---|---|
| Patched | `da0ada979471ced5a1f64c514998f65c206b2276` (this PR) |
| Baseline | `e7d3cb3e3fec8ca8431bf53cfb5edc6cb99febdf` (this PR's parent) |
### Command
```
mvnw.cmd -B -ntp -Duse.platformToolsetVersion=v143 \
-pl hadoop-tools/hadoop-azure -Dparallel-tests=abfs -Dscale verify
```
### Results — patched vs baseline, same VM and same account
| | Patched | Baseline |
|---|---|---|
| Tests run | **1494** | 1489 |
| Failures | **10** | 9 |
| Errors | **22** | 23 |
| Skipped | 474 | 474 |
| Duration | 28:10 | 28:24 |
Both runs exited 0, with 343 ABFS test classes started in each.
**No regression.** The failing sets are identical in both runs except for two
single, retry-flaky entries that cancel each other out:
| Test | Patched | Baseline |
|---|---|---|
| `TestAbfsInputStream.testAdaptiveInputStream` | fails | passes |
| `ITestAzureBlobFileSystemRenameUnicode.testRenameFileUsingUnicode` (retry
5) | passes | fails |
Neither relates to list-status path construction.
### The fix is actually exercised
```
TestAbfsClient: patched 3 tests, 0 failures
baseline 2 tests, 0 failures
```
The additional test is `testGetVersionedFileStatusFromEntryUsesForwardSlash`,
passing on Windows. And the directly relevant integration suite is clean:
```
ITestAzureBlobFileSystemListStatus: Tests run: 20, Failures: 0, Errors: 0,
Skipped: 7
```
### Failures present in both runs (pre-existing / environmental)
| Cause | Tests |
|---|---|
| `There is no primary group for UGI SYSTEM` — tests ran in a service
context without a primary group | `ITestAbfsIdentityTransformer`,
`ITestAzureBlobFileSystemBasics.testWorkingDirectory` |
| 403 `AuthorizationPermissionMismatch` on `setAccessControl` — the service
principal lacks container-root ACL rights |
`ITestAzureBlobFilesystemAcl.testEnsureAclOperationWorksForRoot` |
| OAuth-only account; test expects SharedKey / user-bound SAS to be
configured | `ITestAbfsClient.testAuthTypeProviderSetup` (×3) |
| Pre-existing, unchanged between runs |
`ITestAzureBlobFileSystemDelete.testDeleteIdempotencyTriggerHttp404`,
`ITestAzureBlobFileSystemE2E.testHttpReadTimeout`, `ITestAbfsWrappedIO`
teardown NPE |
| Unit tests (surefire), unchanged between runs, unrelated to ABFS listing |
`TestAggregateMetricsManager` (×3), `TestReadBufferManagerV2` (×2),
`TestShellDecryptionKeyProvider` |
### Separate finding, not related to this PR
Building `hadoop-common` on Windows fails against **Windows SDK 10.0.26100**:
```
winutils.h(129,7): error C2733: 'GetFileInformationByName':
you cannot overload a function with 'extern "C"' linkage
```
Hadoop declares its own `GetFileInformationByName` in
`hadoop-common/src/main/winutils/include/winutils.h`, and a function of the
same
name now exists in the newer SDK. `libwinutils.vcxproj` does not pin
`WindowsTargetPlatformVersion`, so msbuild selects the newest SDK installed.
I
worked around it by pinning SDK 10.0.22621. Happy to raise this as its own
JIRA if
it is not already tracked.
Also worth noting for anyone reproducing this on Build Tools rather than the
full
Visual Studio IDE: `convert-ms-winutils` requires `devenv`, so
`-Duse.platformToolsetVersion=v143` is needed to route through the
`compile-ms-winutils-using-build-tools` path.
### Caveats
1. The suite ran in a service account context, which causes the two "primary
group"
failures above; running as an interactive user would remove them.
2. The account is OAuth-only, so SharedKey and user-bound SAS code paths
were not
covered by this run.
3. The Windows SDK pin affects the build harness only, not the code under
test.
> Hadoop file system can not browse the folder that has children folder with
> colon in Windows
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-19973
> URL: https://issues.apache.org/jira/browse/HADOOP-19973
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.4.3, 3.5.1
> Reporter: Toan Le
> Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.4, 3.5.1
>
> Attachments: Screenshot 2026-02-11 at 11.40.36.png,
> test_hadoop_azure.zip
>
>
> Hi guys,
> I am facing an issue when browsing the children's folder name with a colon in
> ADLS.
> Here is my case. My root folder in ADLS contains a folder with a name that
> includes a colon.
> When I used the Hadoop file system. I got this error on Windows:
> {code:java}
> java -cp ".;./*" AbfsListStatusTest.java [main] WARN
> org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> [main] ERROR org.apache.hadoop.fs.azurebfs.services.AbfsClient - Unable to
> deserialize list results for Uri
> abfss://<containerName>@<storeAccountName>.dfs.core.windows.net
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.Path.initialize(Path.java:264) at
> org.apache.hadoop.fs.Path.<init>(Path.java:222) at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getVersionedFileStatusFromEntry(AbfsClient.java:1846)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1373)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: \FestSpecial:
> at java.base/java.net.URI$Parser.fail(URI.java:2995)
> at java.base/java.net.URI$Parser.checkChars(URI.java:3166)
> at java.base/java.net.URI$Parser.checkChar(URI.java:3176)
> at java.base/java.net.URI$Parser.parse(URI.java:3191)
> at java.base/java.net.URI.<init>(URI.java:807)
> at org.apache.hadoop.fs.Path.initialize(Path.java:261)
> ... 14 more Exception in thread "main" Status code: -1 error code: null error
> message: Parsing of Json List Response Failed in
> DfsClient.java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Illegal character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1390)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45) Caused
> by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0: \FestSpecial: at
> org.apache.hadoop.fs.Path.initialize(Path.java:264) at
> org.apache.hadoop.fs.Path.<init>(Path.java:222) at
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getVersionedFileStatusFromEntry(AbfsClient.java:1846)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.parseListPathResults(AbfsDfsClient.java:1373)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsDfsClient.listPath(AbfsDfsClient.java:347)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1278)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1242)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:1224)
> at
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:713)
> at AbfsListStatusOAuthTest.main(AbfsListStatusTest.java:45)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.execute(Main.java:484)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.run(Main.java:208)
> at jdk.compiler/com.sun.tools.javac.launcher.Main.main(Main.java:135)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: \FestSpecial: at
> java.base/java.net.URI$Parser.fail(URI.java:2995) at
> java.base/java.net.URI$Parser.checkChars(URI.java:3166) at
> java.base/java.net.URI$Parser.checkChar(URI.java:3176) at
> java.base/java.net.URI$Parser.parse(URI.java:3191) at
> java.base/java.net.URI.<init>(URI.java:807) at
> org.apache.hadoop.fs.Path.initialize(Path.java:261) ... 14 more
> {code}
> I tested my script, and it worked on Linux.
> Could you please help me check it?
> I compressed all my scripts ({_}AbfsListStatusTest.java{_}) and library into
> the ticket ({_}test_hadoop_azure.zip{_}).
> Here is the _AbfsListStatusTest.java:_
> {code:java}
> {code}
> _import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileStatus;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import java.net.URI;
> public class AbfsListStatusOAuthTest {
> public static void main(String[] args) throws Exception {
> // Inputs (use env vars or hardcode)
> String accountName = System.getenv("ADLS_STORAGE_ACCOUNT_NAME");
> String containerName = System.getenv("ADLS_CONTAINER_NAME");
> String tenantId = System.getenv("ADLS_TENANT_ID");
> String clientId = System.getenv("ADLS_CLIENT_ID");
> String clientSecret = System.getenv("ADLS_CLIENT_SECRET");
> String dirPath = (args.length > 0) ? args[0] : "/";
> // Use abfss for OAuth
> String fsUri = String.format("abfss://%s@%s.dfs.core.windows.net",
> containerName, accountName);
> String accountFqdn = accountName + ".dfs.core.windows.net";
> Configuration conf = new Configuration(false);
> // Ensure ABFS impls are known
> conf.set("fs.abfss.impl",
> "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem");
> conf.set("fs.abfs.impl", "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem");
> // — OAuth configs (scoped to the account host) —
> conf.set("fs.azure.account.auth.type." + accountFqdn, "OAuth");
> conf.set("fs.azure.account.oauth.provider.type." + accountFqdn,
> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider");
> conf.set("fs.azure.account.oauth2.client.id." + accountFqdn, clientId);
> conf.set("fs.azure.account.oauth2.client.secret." + accountFqdn,
> clientSecret);
> // Token endpoint (try v1 first — Hadoop ABFS commonly uses this)
> conf.set("fs.azure.account.oauth2.client.endpoint." + accountFqdn,
> "https://login.microsoftonline.com/"; + tenantId + "/oauth2/token");
> // Create FS and call listStatus
> FileSystem fs = FileSystem.get(new URI(fsUri), conf);
> FileStatus[] statuses = fs.listStatus(new Path(dirPath));
> System.out.println("Found " + statuses.length + " entries under " + dirPath);
> for (FileStatus st : statuses)
> { System.out.printf("%s isDir=%s len=%d%n", st.getPath(), st.isDirectory(),
> st.getLen()); }
> fs.close();
> }
> }_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
