[ https://issues.apache.org/jira/browse/HADOOP-17611?focusedWorklogId=582333&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-582333 ]
ASF GitHub Bot logged work on HADOOP-17611: ------------------------------------------- Author: ASF GitHub Bot Created on: 14/Apr/21 10:03 Start Date: 14/Apr/21 10:03 Worklog Time Spent: 10m Work Description: amaroti edited a comment on pull request #2897: URL: https://github.com/apache/hadoop/pull/2897#issuecomment-819387790 @bgaborg I have tested this manully by hand using two clusters. I have not yet looked into how the unit tests look like for hadoop. I will take a look at it. Also Ayush Saxena (@ayushtkn) had made some usefull points on the jira ticket that I will take a look at shortly: https://issues.apache.org/jira/browse/HADOOP-17611?focusedCommentId=17320445&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17320445 > Seems there are two PRs, More or less doing the same thing I guess, I just had a glance on the second one, > > So, who ever plans to chase this, Couple of points to keep in mind: > > We need a test in AbstractContractDistCpTest which all the FileSystems can also use > Should cover two scenarios. First When preserve Time is specified it should preserve time and when not it shouldn't in case of parallel copy. The latter case is working ok as of now, To make sure we don't change the behaviour > The parent modification time is to be preserved when the parent is in the scope of copy, not always. say your are copying /dir/fil1 to /dir1/file2 using parallel copy, then we don't touch /dir1 AFAIK > > The above are the basic requirements, Now the below stuff, If possible we should do: > > For parent directories preserve only once, say if you have 10K files under that parent, then do that setTimes 10K times. > And if the parallel copy is enabled, there is no point of preserving before concat operation, we can save that call. > > > > This isn't a one liner, and throw some challenges, So, please decide who wants to chase this and together work on one PR only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 582333) Time Spent: 2h 50m (was: 2h 40m) > Distcp parallel file copy breaks the modification time > ------------------------------------------------------ > > Key: HADOOP-17611 > URL: https://issues.apache.org/jira/browse/HADOOP-17611 > Project: Hadoop Common > Issue Type: Bug > Reporter: Adam Maroti > Assignee: Adam Maroti > Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > The commit HADOOP-11794. Enable distcp to copy blocks in parallel. > (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of > large files. > > In CopyCommitter.java inside concatFileChunks Filesystem.concat is called > which changes the modification time therefore the modification times of files > copeid by distcp will not match the source files. However this only occurs > for large enough files, which are copied by splitting them up by distcp. > In concatFileChunks before calling concat extract the modification time and > apply that to the concatenated result-file after the concat. (probably best > -after- before the rename()). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org