OneSizeFitsQuorum opened a new pull request, #8858: URL: https://github.com/apache/hadoop/pull/8858
### Description of PR [HADOOP-17611](https://issues.apache.org/jira/browse/HADOOP-17611) With `-blocksperchunk` and `-pt`, CopyMapper preserves timestamps on the chunks, but the subsequent concat in CopyCommitter changes the final file's modification time. Restore the source modification and access times after concatenation, rename, and checksum validation, only when `FileAttribute.TIMES` is requested. Restore only timestamps: file entries in the copy listing do not contain extended ACLs, so replaying all preserved attributes here can erase ACLs already preserved by the mapper. Directory preservation continues to use the existing commit phase and copy scope. Add 14 parameterized cases to AbstractContractDistCpTest covering: - TIMES enabled and disabled with ordinary, direct, and atomic copies. - Actual multi-block files, nested directories, exact-block files, small files, and empty files, with complete content verification. - Parents outside the copy scope, `-updateRoot`, overwritten files, and fully skipped updates. - ACLs, XAttrs, permissions, ownership, replication, and block size alongside optional timestamp preservation. The tests use two concurrent local map tasks and assert that large source files really span multiple blocks. Capability checks skip unsupported filesystems. ### How was this patch tested? JDK 17; local Maven reactor run: ```sh mvn -B -pl hadoop-tools/hadoop-distcp -am \ -Dtest=TestHDFSContractDistCp,TestLocalContractDistCp,TestDistCpSystem,TestCopyMapper,TestCopyCommitter,TestDistCpWithAcls,TestDistCpWithXAttrs \ -Dmaven.test.failure.ignore=false \ -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \ -Dcheckstyle.skip -Drat.skip=true test ``` 99 tests: 83 passed, 16 skipped, zero failures or errors. All 14 new HDFS cases passed. The skips are the 14 unsupported local-filesystem cases and two existing disabled HDFS tests. The run used a temporary Maven repository and mirror settings. Separately ran the repository's Checkstyle configuration on both changed files: no findings on added lines. `git diff --check` passed. Also ran temporary negative controls against the same contract tests: the original implementation fails the three TIMES-enabled copy-mode cases; unconditional mtime/atime preservation fails the TIMES-disabled cases; preserving an out-of-scope parent fails scope checks; and replaying all attributes fails the ACL checks. Scope note: ordinary chunked single-file copying to a nonexistent destination filename has a separate staging-path failure (`Target is dir, Source is file`), reproduced with both the original and patched implementations. Scope tests cover ordinary file-to-existing-directory copying and direct file-to-new-filename copying. This PR does not change staging paths or optimize mapper-side timestamp calls. ### For code changes: - [x] The title starts with the corresponding JIRA issue ID. - Object storage integration tests: not run; validation used HDFS and local filesystem contracts. - New dependencies: none. - LICENSE/NOTICE changes: not applicable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
