vt921102 commented on PR #8693:
URL: https://github.com/apache/hadoop/pull/8693#issuecomment-6118911789
Thanks @anujmodi2021 — integration validation for `branch-3.4` is complete.
As with #8692 I ran this **on Windows**, and for this branch also **on JDK
8**:
`branch-3.4` is the last Hadoop line supporting Java 8, and the defect only
reproduces where `File.separator` is a backslash. So this exercises both the
target
platform and the target JDK for this backport.
### Environment
- Windows 11 Pro 24H2, 4 vCPU / 8 GiB
- **Eclipse Temurin JDK 1.8.0_504**, Apache Maven 3.9.16
- Visual Studio 2022 Build Tools, MSVC v143, Windows SDK 10.0.22621
- ADLS Gen2 (StorageV2, Standard_LRS), **HNS enabled**, **North Europe**,
auth = **SharedKey**
The storage account belongs to my employer, so I have not named it here;
happy to
share it with a committer privately if the project needs it on record.
### Commits tested
| Role | Commit |
|---|---|
| Patched | `28e36e8de29ecc7f2656f2589863915cbc029ddd` (this PR) |
| Baseline | `536b278f3eae32707dae7f9d73b55c254d58b535` (this PR's parent) |
### Command
```
mvn.cmd -B -ntp -Duse.platformToolsetVersion=v143 \
-pl hadoop-tools/hadoop-azure -Dparallel-tests=abfs -Dscale verify
```
### Results — patched vs baseline, same VM and account
| | Patched | Baseline |
|---|---|---|
| Tests run | **1531** | 1530 |
| Failures | **8** | 7 |
| Errors | **45** | 46 |
| Skipped | 260 | 260 |
| Duration | 14:23 | 16:01 |
| Unique failing entries | **39** | **39** |
Both exited 0, with 139 ABFS test classes started in each.
**No regression.** 38 of the 39 failing entries are identical. The two
differences
are retry-flaky and cancel each other out:
| Test | Patched | Baseline |
|---|---|---|
| `TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting` |
fails | passes |
| `ITestAzureBlobFileSystemRenameUnicode.testRenameFileUsingUnicode` |
passes | fails |
The first is a timing-based fuzzy-validation unit test; the second also
flaked in the
`branch-3.5` baseline. Neither relates to list-status path construction.
### The JUnit 4 adaptation works
```
TestAbfsClient: patched 3 tests, 0 failures
baseline 2 tests, 0 failures
```
The additional test is
`testGetVersionedFileStatusFromEntryUsesForwardSlash`. This is
the one place this backport differs from trunk — `branch-3.4` is still on
JUnit 4,
its `AbfsDfsClient` constructor takes six arguments and its
`AbfsClientContextBuilder`
has no `withFileSystemId` — so confirming it compiles and passes under JDK 8
is the
point of this run. The directly relevant suite is also clean in both runs:
```
ITestAzureBlobFileSystemListStatus: Tests run: 20, Failures: 0, Errors: 0,
Skipped: 7
```
### Failures present in both runs
| Cause | Tests |
|---|---|
| Account is SharedKey-only; these tests need a SAS token provider
configured | `ITestAzureBlobFileSystemAuthorization` (~26),
`ITestAzureBlobFileSystemChooseSAS` (3) |
| Ran in a service account context with no primary group |
`ITestAbfsIdentityTransformer`,
`ITestAzureBlobFileSystemBasics.testWorkingDirectory` |
| YARN mini-cluster does not start on this Windows host |
`ITestAbfsTerasort` |
| Pre-existing, unchanged between runs |
`ITestAzureBlobFileSystemDelete.testDeleteIdempotencyTriggerHttp404` |
### Notes for anyone reproducing this on `branch-3.4`
1. `branch-3.4` has no `mvnw.cmd`, so Maven must be installed separately.
2. On Build Tools rather than the full VS IDE,
`-Duse.platformToolsetVersion=v143`
is needed because `convert-ms-winutils` requires `devenv`.
3. Windows SDK must be pinned to 10.0.22621; winutils does not compile
against
10.0.26100 (`error C2733: 'GetFileInformationByName'`). I mentioned this
on #8692
and am happy to file it as its own JIRA.
### Caveats
1. SharedKey-only account, so SAS and OAuth code paths were not covered by
this run;
that accounts for most of the errors seen identically in both runs.
2. The suite ran in a service account context, causing the two failures
noted above.
3. The SDK pin affects the build harness only, not the code under test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
