[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347036#comment-14347036
]
Hudson commented on HDFS-7682:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346999#comment-14346999
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346933#comment-14346933
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346924#comment-14346924
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346719#comment-14346719
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/856/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346709#comment-14346709
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346259#comment-14346259
]
Hudson commented on HDFS-7682:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7250 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7250/])
HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file
includes non-snapshotted content. Contributed by Charles Lamb. (atm: rev
f2d7a67a2c1d9dde10ed3171fdec65dff885afcc)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotFileLength.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Fix For: 2.7.0
>
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344483#comment-14344483
]
Jing Zhao commented on HDFS-7682:
-
+1 from me too.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344305#comment-14344305
]
Aaron T. Myers commented on HDFS-7682:
--
Latest patch looks good to me, and seems like it should address [~jingzhao]'s
comments.
[~jingzhao] - does the patch look good to you as well? If so, I'll be +1 on it.
Thanks gents.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316843#comment-14316843
]
Hadoop QA commented on HDFS-7682:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698108/HDFS-7682.003.patch
against trunk revision b94c111.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9540//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9540//console
This message is automatically generated.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch, HDFS-7682.003.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308012#comment-14308012
]
Jing Zhao commented on HDFS-7682:
-
Thanks Charles. My main concern is that the patch is just a partial fix, since
it cannot cover the case that the file is snapshotted but still being written.
bq. In other words, the behavior for non-snapshotted files that are still open
(and possibly being appended to) is not changed by this patch, only that of
snapshotted files, for which isLastBlockComplete() is a valid check.
The behavior for snapshotted files that are still open also have not been
changed.
Actually for a snapshotted file, {{blockLocations.getFileLength}} should equal
to the file length explicitly recorded in the snapshot diff. If there is not
such length recorded, {{blockLocations.getFileLength}} should be the current
file length including the last uc block's length (please read the current code
to confirm). In that case, the check condition should be "if the src is a
snapshot path", and we should use {{blockLocations.getFileLength}} as the limit.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307935#comment-14307935
]
Hadoop QA commented on HDFS-7682:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696791/HDFS-7682.002.patch
against trunk revision afbecbb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9441//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9441//console
This message is automatically generated.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch,
> HDFS-7682.002.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301885#comment-14301885
]
Charles Lamb commented on HDFS-7682:
[~jingzhao],
Did you have any more comments on this Jira?
BTW, the test failure is unrelated.
Thanks.
Charles
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294146#comment-14294146
]
Hadoop QA commented on HDFS-7682:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694806/HDFS-7682.001.patch
against trunk revision f56da3c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9347//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9347//console
This message is automatically generated.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch, HDFS-7682.001.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292763#comment-14292763
]
Jing Zhao commented on HDFS-7682:
-
Thanks for working on this, [~clamb].
One question about the current patch. The following code means we only do the
length checking if the file is complete. Then for a snapshotted while still
being written file, we will still have the issue. How about changing the
condition to "if the src is a snapshot path"? Then we can use
"{{blockLocations.getFileLength}} + {{last block's length if it's incomplete}}"
as the length limit.
{code}
+if (blockLocations.isLastBlockComplete()) {
+ remaining = Math.min(length, blockLocations.getFileLength());
+}
{code}
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7682) {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
[
https://issues.apache.org/jira/browse/HDFS-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292659#comment-14292659
]
Hadoop QA commented on HDFS-7682:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694609/HDFS-7682.000.patch
against trunk revision 21d5599.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9330//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9330//console
This message is automatically generated.
> {{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes
> non-snapshotted content
>
>
> Key: HDFS-7682
> URL: https://issues.apache.org/jira/browse/HDFS-7682
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-7682.000.patch
>
>
> DistributedFileSystem#getFileChecksum of a snapshotted file includes
> non-snapshotted content.
> The reason why this happens is because DistributedFileSystem#getFileChecksum
> simply calculates the checksum of all of the CRCs from the blocks in the
> file. But, in the case of a snapshotted file, we don't want to include data
> in the checksum that was appended to the last block in the file after the
> snapshot was taken.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
