[
https://issues.apache.org/jira/browse/HADOOP-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130774#comment-18130774
]
ASF GitHub Bot commented on HADOOP-17611:
-----------------------------------------
OneSizeFitsQuorum opened a new pull request, #8858:
URL: https://github.com/apache/hadoop/pull/8858
### Description of PR
[HADOOP-17611](https://issues.apache.org/jira/browse/HADOOP-17611)
With `-blocksperchunk` and `-pt`, CopyMapper preserves timestamps on the
chunks, but the subsequent concat in CopyCommitter changes the final file's
modification time. Restore the source modification and access times after
concatenation, rename, and checksum validation, only when `FileAttribute.TIMES`
is requested.
Restore only timestamps: file entries in the copy listing do not contain
extended ACLs, so replaying all preserved attributes here can erase ACLs
already preserved by the mapper. Directory preservation continues to use the
existing commit phase and copy scope.
Add 14 parameterized cases to AbstractContractDistCpTest covering:
- TIMES enabled and disabled with ordinary, direct, and atomic copies.
- Actual multi-block files, nested directories, exact-block files, small
files, and empty files, with complete content verification.
- Parents outside the copy scope, `-updateRoot`, overwritten files, and
fully skipped updates.
- ACLs, XAttrs, permissions, ownership, replication, and block size
alongside optional timestamp preservation.
The tests use two concurrent local map tasks and assert that large source
files really span multiple blocks. Capability checks skip unsupported
filesystems.
### How was this patch tested?
JDK 17; local Maven reactor run:
```sh
mvn -B -pl hadoop-tools/hadoop-distcp -am \
-Dtest=TestHDFSContractDistCp,TestLocalContractDistCp,TestDistCpSystem,TestCopyMapper,TestCopyCommitter,TestDistCpWithAcls,TestDistCpWithXAttrs
\
-Dmaven.test.failure.ignore=false \
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false \
-Dcheckstyle.skip -Drat.skip=true test
```
99 tests: 83 passed, 16 skipped, zero failures or errors. All 14 new HDFS
cases passed. The skips are the 14 unsupported local-filesystem cases and two
existing disabled HDFS tests. The run used a temporary Maven repository and
mirror settings.
Separately ran the repository's Checkstyle configuration on both changed
files: no findings on added lines. `git diff --check` passed.
Also ran temporary negative controls against the same contract tests: the
original implementation fails the three TIMES-enabled copy-mode cases;
unconditional mtime/atime preservation fails the TIMES-disabled cases;
preserving an out-of-scope parent fails scope checks; and replaying all
attributes fails the ACL checks.
Scope note: ordinary chunked single-file copying to a nonexistent
destination filename has a separate staging-path failure (`Target is dir,
Source is file`), reproduced with both the original and patched
implementations. Scope tests cover ordinary file-to-existing-directory copying
and direct file-to-new-filename copying. This PR does not change staging paths
or optimize mapper-side timestamp calls.
### For code changes:
- [x] The title starts with the corresponding JIRA issue ID.
- Object storage integration tests: not run; validation used HDFS and local
filesystem contracts.
- New dependencies: none.
- LICENSE/NOTICE changes: not applicable.
> Distcp parallel file copy breaks the modification time
> ------------------------------------------------------
>
> Key: HADOOP-17611
> URL: https://issues.apache.org/jira/browse/HADOOP-17611
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Adam Maroti
> Assignee: Adam Maroti
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The commit HADOOP-11794. Enable distcp to copy blocks in parallel.
> (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of
> large files.
>
> In CopyCommitter.java inside concatFileChunks Filesystem.concat is called
> which changes the modification time therefore the modification times of files
> copeid by distcp will not match the source files. However this only occurs
> for large enough files, which are copied by splitting them up by distcp.
> In concatFileChunks before calling concat extract the modification time and
> apply that to the concatenated result-file after the concat. (probably best
> -after- before the rename()).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
