[ 
https://issues.apache.org/jira/browse/HDFS-17120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747480#comment-17747480
 ] 

ASF GitHub Bot commented on HDFS-17120:
---------------------------------------

hadoop-yetus commented on PR #5885:
URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1651686956

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 29s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  31m 35s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 26s |  |  trunk passed with JDK 
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1  |
   | +1 :green_heart: |  compile  |   0m 24s |  |  trunk passed with JDK 
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09  |
   | +1 :green_heart: |  checkstyle  |   0m 24s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 28s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 28s |  |  trunk passed with JDK 
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1  |
   | +1 :green_heart: |  javadoc  |   0m 24s |  |  trunk passed with JDK 
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09  |
   | +1 :green_heart: |  spotbugs  |   0m 40s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  20m 39s |  |  branch has no errors 
when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 18s |  |  the patch passed with JDK 
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1  |
   | +1 :green_heart: |  javac  |   0m 18s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 16s |  |  the patch passed with JDK 
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09  |
   | +1 :green_heart: |  javac  |   0m 16s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 13s | 
[/results-checkstyle-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5885/8/artifact/out/results-checkstyle-hadoop-tools_hadoop-distcp.txt)
 |  hadoop-tools/hadoop-distcp: The patch generated 1 new + 20 unchanged - 0 
fixed = 21 total (was 20)  |
   | +1 :green_heart: |  mvnsite  |   0m 18s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 16s |  |  
hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 with 
JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 generated 0 new + 37 unchanged 
- 4 fixed = 37 total (was 41)  |
   | +1 :green_heart: |  javadoc  |   0m 16s |  |  
hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
 with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 0 
new + 37 unchanged - 4 fixed = 37 total (was 41)  |
   | +1 :green_heart: |  spotbugs  |   0m 37s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 55s |  |  patch has no errors 
when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  13m 20s |  |  hadoop-distcp in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 29s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   |  96m 22s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5885/8/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5885 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux b944c168be7f 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 55d1196a354d68045d8d2452eb722671928a3d65 |
   | Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5885/8/testReport/ |
   | Max. process+thread count | 560 (vs. ulimit of 5500) |
   | modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5885/8/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   




> Support snapshot diff based copylisting for flat paths.
> -------------------------------------------------------
>
>                 Key: HDFS-17120
>                 URL: https://issues.apache.org/jira/browse/HDFS-17120
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Sadanand Shenoy
>            Assignee: Sadanand Shenoy
>            Priority: Major
>              Labels: pull-request-available
>
> Currently for Diff-based copyListing that is used during the distcpSync step 
> of an incremental copy by default the SimpleCopyListing implementation is 
> used.  In it's implementation it iterates through the DiffReport and if the 
> DiffType is Create and the path is a directory, it recursively traverses the 
> directory and adds the subpaths to the resultant copyList.
> This works fine for implementations of snapshotDiff that include only 
> top-level directories as part of its DiffReport . Suppose a snapshotDiff 
> implementation outputs only flat paths that include both the directory and 
> sub-directory subpath in its DiffReport, it will lead to duplicate paths in 
> the copyList and throws DuplicateFileException.
>  
> For example
> Ozone filesystem implementation of snapdiff b/w 2 snapshots shows all 
> subpaths as part of the diff.
> {code:java}
> [~]# ozone sh snapshot create vol11/buck1 snap1
> [~]# ozone sh snapshot create vol11/buck2 snap1
> [~]# ozone fs -mkdir ofs://ozone1/vol11/buck1/dir1
> [ ~]# ozone fs -mkdir ofs://ozone1/vol11/buck1/dir1/dir11
> [ ~]# ozone fs -mkdir ofs://ozone1/vol11/buck1/dir1/dir11/dir111
> [~]# ozone sh snapshot create vol11/buck1 snap2 
> [~]# ozone sh snapshot diff vol11/buck1 snap1 snap2
> Difference between snapshot: snap1 and snapshot: snap2
> +    ./dir1
> +    ./dir1/dir11
> +    ./dir1/dir11/dir111  {code}
> we can see even though dir11 & dir111 are subpaths they are present in 
> snapdiff , This is not the case for HDFS though.
> This Jira aims to create a new copyListing impl that is used for diff based 
> copyListing that doesn't traverse the directory but only adds paths that are 
> present in its diff.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to