[ https://issues.apache.org/jira/browse/HDFS-12618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207768#comment-16207768 ]
Hadoop QA commented on HDFS-12618: ---------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 91 unchanged - 4 fixed = 94 total (was 95) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 28s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}139m 17s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0de40f0 | | JIRA Issue | HDFS-12618 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12892595/HDFS-121618.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3f5ab7aa89b6 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 31ebccc | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21726/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21726/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21726/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21726/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > fsck -includeSnapshots reports wrong amount of total blocks > ----------------------------------------------------------- > > Key: HDFS-12618 > URL: https://issues.apache.org/jira/browse/HDFS-12618 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools > Affects Versions: 3.0.0-alpha3 > Reporter: Wellington Chevreuil > Assignee: Wellington Chevreuil > Priority: Minor > Attachments: HDFS-121618.001.patch, HDFS-121618.initial > > > When snapshot is enabled, if a file is deleted but is contained by a > snapshot, *fsck* will not reported blocks for such file, showing different > number of *total blocks* than what is exposed in the Web UI. > This should be fine, as *fsck* provides *-includeSnapshots* option. The > problem is that *-includeSnapshots* option causes *fsck* to count blocks for > every occurrence of a file on snapshots, which is wrong because these blocks > should be counted only once (for instance, if a 100MB file is present on 3 > snapshots, it would still map to one block only in hdfs). This causes fsck to > report much more blocks than what actually exist in hdfs and is reported in > the Web UI. > Here's an example: > 1) HDFS has two files of 2 blocks each: > {noformat} > $ hdfs dfs -ls -R / > drwxr-xr-x - root supergroup 0 2017-10-07 21:21 /snap-test > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 /snap-test/file1 > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 /snap-test/file2 > drwxr-xr-x - root supergroup 0 2017-05-13 13:03 /test > {noformat} > 2) There are two snapshots, with the two files present on each of the > snapshots: > {noformat} > $ hdfs dfs -ls -R /snap-test/.snapshot > drwxr-xr-x - root supergroup 0 2017-10-07 21:21 > /snap-test/.snapshot/snap1 > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 > /snap-test/.snapshot/snap1/file1 > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 > /snap-test/.snapshot/snap1/file2 > drwxr-xr-x - root supergroup 0 2017-10-07 21:21 > /snap-test/.snapshot/snap2 > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:16 > /snap-test/.snapshot/snap2/file1 > -rw-r--r-- 1 root supergroup 209715200 2017-10-07 20:17 > /snap-test/.snapshot/snap2/file2 > {noformat} > 3) *fsck -includeSnapshots* reports 12 blocks in total (4 blocks for the > normal file path, plus 4 blocks for each snapshot path): > {noformat} > $ hdfs fsck / -includeSnapshots > FSCK started by root (auth:SIMPLE) from /127.0.0.1 for path / at Mon Oct 09 > 15:15:36 BST 2017 > Status: HEALTHY > Number of data-nodes: 1 > Number of racks: 1 > Total dirs: 6 > Total symlinks: 0 > Replicated Blocks: > Total size: 1258291200 B > Total files: 6 > Total blocks (validated): 12 (avg. block size 104857600 B) > Minimally replicated blocks: 12 (100.0 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 0 (0.0 %) > Mis-replicated blocks: 0 (0.0 %) > Default replication factor: 1 > Average block replication: 1.0 > Missing blocks: 0 > Corrupt blocks: 0 > Missing replicas: 0 (0.0 %) > {noformat} > 4) Web UI shows the correct number (4 blocks only): > {noformat} > Security is off. > Safemode is off. > 5 files and directories, 4 blocks = 9 total filesystem object(s). > {noformat} > I would like to work on this solution, will propose an initial solution > shortly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org