[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126539#comment-14126539 ] Haohui Mai commented on HDFS-6981: -- +1 > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, > HDFS-6981.06.patch, HDFS-6981.07.patch, HDFS-6981.08.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126480#comment-14126480 ] Arpit Agarwal commented on HDFS-6981: - All test failures look unrelated? > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, > HDFS-6981.06.patch, HDFS-6981.07.patch, HDFS-6981.08.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126424#comment-14126424
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667230/HDFS-6981.08.patch
against trunk revision d989ac0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZones
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7954//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7954//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch,
> HDFS-6981.06.patch, HDFS-6981.07.patch, HDFS-6981.08.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126011#comment-14126011
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667061/HDFS-6981.07.patch
against trunk revision 302d9a0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7950//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7950//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7950//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch,
> HDFS-6981.06.patch, HDFS-6981.07.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125200#comment-14125200
]
Arpit Agarwal commented on HDFS-6981:
-
bq. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 2.0.3)
warnings.
I think Jenkins is hitting a bug. findbugs passed for me locally and the link
to the warnings is broken.
{color:green}+1 overall{color}.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version ) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch,
> HDFS-6981.06.patch, HDFS-6981.07.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125108#comment-14125108
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667061/HDFS-6981.07.patch
against trunk revision a23144f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7944//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7944//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7944//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch,
> HDFS-6981.06.patch, HDFS-6981.07.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124958#comment-14124958 ] James Thomas commented on HDFS-6981: +1, looks good to me. Getting a 404 when I try to look at the Findbugs warnings -- any idea what's causing those? > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, > HDFS-6981.06.patch, HDFS-6981.07.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124785#comment-14124785
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667061/HDFS-6981.07.patch
against trunk revision 88209ce.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7932//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7932//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7932//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch,
> HDFS-6981.06.patch, HDFS-6981.07.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124731#comment-14124731
]
James Thomas commented on HDFS-6981:
{code}
+ if (!storagesWithRollingUpgradeMarker.contains(bpRoot.toString()) &&
+ !markerFile.exists()) {
+LOG.info("Created " + markerFile);
+markerFile.createNewFile();
+storagesWithRollingUpgradeMarker.add(bpRoot.toString());
+storagesWithoutRollingUpgradeMarker.remove(bpRoot.toString());
+ }
{code}
could be
{code}
+ if (!storagesWithRollingUpgradeMarker.contains(bpRoot.toString())) {
+if (!markerFile.exists()) {
+ LOG.info("Created " + markerFile);
+ markerFile.createNewFile();
+ storagesWithRollingUpgradeMarker.add(bpRoot.toString());
+ storagesWithoutRollingUpgradeMarker.remove(bpRoot.toString());
+} else {
+ storagesWithRollingUpgardeMarker.add(bpRoot.toString());
+}
+ }
{code}
and similarly for {{clearRollingUpgradeMarkers}}. These changes ensure that the
cache is in sync with the filesystem state and reduce the number of filesystem
operations.
It also seems to me like the in-memory cache could be just be two volatile
booleans (e.g. {{storagesHaveRollingUpgradeMarker}} and
{{storagesDoNotHaveRollingUpgradeMarker}}) rather than two sets. Could the set
of storages possibly change during the rolling upgrade?
Otherwise things look good. Tests are solid.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, HDFS-6981.06.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124602#comment-14124602 ] Arpit Agarwal commented on HDFS-6981: - Latest test failures are unrelated to the patch. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, HDFS-6981.06.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124342#comment-14124342
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666983/HDFS-6981.06.patch
against trunk revision e6420fe.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7928//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7928//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7928//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch, HDFS-6981.06.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124258#comment-14124258
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666923/HDFS-6981.05.patch
against trunk revision 21c0cde.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7923//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7923//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7923//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch, HDFS-6981.05.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123471#comment-14123471 ] James Thomas commented on HDFS-6981: The marker file sounds like the best solution to me. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123413#comment-14123413 ] Arpit Agarwal commented on HDFS-6981: - Lacking an explicit finalize command for rolling upgrade, it is hard for the DN to determine when to delete 'previous'. Rolling upgrade is signaled by the presence/absence of RollingUpgradeInfo in the heartbeat response. Without modifying the NN, one solution is that the DN creates a marker file when rolling upgrade is signaled by NN. When rolling upgrade is no longer signaled by NN, 'previous' is cleaned up only if the marker file is present. Else a regular upgrade is in progress and 'previous' is left alone. I am wary of making NN changes, the interaction with HA is complex enough as it is. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122555#comment-14122555 ] Arpit Agarwal commented on HDFS-6981: - So removing 'previous' after a rolling upgrade is finalized is not easy to solve, since there is no explicit finalize command. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122513#comment-14122513
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1256/HDFS-6981.04.patch
against trunk revision 6104520.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSUpgrade
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7907//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7907//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122384#comment-14122384 ] Arpit Agarwal commented on HDFS-6981: - I think that will be fine - heartbeats are sent every 3 seconds by default. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch, HDFS-6981.04.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122378#comment-14122378
]
James Thomas commented on HDFS-6981:
bq. Not an issue. If there was no layout version change then 'previous' will be
absent and doTransition will skip calling doRollback, instead restoring from
trash. An existing test case already verifies it by starting the DN with the
-rollback flag.
Right, I am talking about the case where we roll back to a version of the
software prior to HDFS-6800, in which case the following code will be executed
in {{doTransition}}:
{code}
if (startOpt == StartupOption.ROLLBACK) {
doRollback(sd, nsInfo); // rollback if applicable
} else {
// Restore all the files in the trash. The restored files are retained
// during rolling upgrade rollback. They are deleted during rolling
// upgrade downgrade.
int restored = restoreBlockFilesFromTrash(getTrashRootDir(sd));
LOG.info("Restored " + restored + " block files from trash.");
}
{code}
So the trash won't be restored until the first heartbeat response is received
from the NN, at which point {{signalRollingUpgrade}} will be called with
{{null}}, and we'll call {{restoreTrash}}. So the blocks in trash will
potentially be missing for a short period of time -- not sure if this is an
issue.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch, HDFS-6981.04.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122268#comment-14122268
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1225/HDFS-6981.03.patch
against trunk revision f7df24b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7905//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7905//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122262#comment-14122262
]
James Thomas commented on HDFS-6981:
[~arpitagarwal], thanks for the patch. A couple high-level comments:
* The {{Preconditions.checkState}} calls can fail. Consider the case where a
block {{B1}} is created after the rolling upgrade is started and is deleted
before the upgrade is finalized. Since your code avoids adding a block to the
trash only if the block exists in the {{previous}} directory, {{B1}} will be
added to the trash. So both the trash and the {{previous}} directory will exist
concurrently. This behavior is not incorrect (the same block does not exist in
both the trash and {{previous}}), but it will cause the
{{Preconditions.checkState}} calls to fail. It seems like you can either remove
these calls or avoid adding a block to the trash if any {{previous}} directory
exists at all, rather than checking whether the block exists in {{previous}}.
* Suppose that we finalize a rolling upgrade with layout version change at time
{{T1}}, and then start another rolling upgrade with layout version change soon
afterwards at time {{T2}}. Suppose also that there is no block report between
{{T1}} and {{T2}}, so the old {{previous}} directory is not deleted. Now a
block {{B2}} existing in the old {{previous}} is deleted, and it is not added
to the trash by the rules in this patch. Next, this DN is restarted, the old
{{previous}} directory is deleted, and a new {{previous}} directory without
{{B2}} is created, and we have lost {{B2}}. This is admittedly an extreme
corner case, but I think that the existence of {{previous}} for some time after
finalization is somewhat worrisome. It's probably a good idea to rename
{{previous}} to something else (e.g. {{finalized.tmp}}) when
{{signalRollingUpgrade}} is called with null, and then actually do the deletion
asynchronously in the background.
{code}
+ // Delete the second file. Ensure that its block file is in previous.
+ File blockFile1 = getBlockForFile(paths[1], true);
+ fs.delete(paths[1], false);
+ ensureBlockFileInPrevious(blockFile1);
{code}
Might want to assert that this file doesn't exist in trash.
Another issue with this set of changes is that if we call roll back to an old
DN software version with {{-rollback}} and the upgrade did not involve a layout
version change, we'll actually fail to restore the trash and there could be
missing blocks as a result. I'm not sure what the best way to handle this is
... we might need to instruct administrators to use {{-rollback}} only when
rolling back after an upgrade involving a DN layout version change.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch,
> HDFS-6981.03.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122205#comment-14122205 ] Arpit Agarwal commented on HDFS-6981: - Also the override was avoided by adding the new layout version to DataNodeLayoutVersion.FEATURES directly. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch, > HDFS-6981.03.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121945#comment-14121945
]
Arpit Agarwal commented on HDFS-6981:
-
Thanks for reviewing Haohui. Let me look into reflection as you suggested.
Overriding versionSupportsFederation is needed so DN storages avoid calling
{{LayoutVersion.supports}}, the override calls
{{DataNodeLayoutVersion.supports}} instead.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121921#comment-14121921
]
Haohui Mai commented on HDFS-6981:
--
{code}
+ /**
+ * Current layout version for DataNode.
+ * Please see {@link DataNodeLayoutVersion.Feature} on adding new layout
version.
+ */
+ public static int getDatanodeLayoutVersion() {
+return DataNodeLayoutVersion.getCurrentLayoutVersion();
+ }
{code}
To avoid the refactoring, you can use reflection to change the value of
{{DATANODE_LAYOUT_VERSION}} to inject new layout version in the test. This
should simplify the patch quite a bit.
{code}
/**
+ * If the DN layout version has been overridden
+ * @param map
+ * @return
+ */
+ @Override
+ public boolean versionSupportsFederation(
+ Map> map,
+ int layoutVersionIn) {
+return DataNodeLayoutVersion.supports(
+LayoutVersion.Feature.FEDERATION, layoutVersionIn);
+ }
+
{code}
It does not seem necessary. Is it related to the above issue?
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120995#comment-14120995
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666409/HDFS-6981.02.patch
against trunk revision 8f1a668.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7893//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7893//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch, HDFS-6981.02.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[
https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120894#comment-14120894
]
Hadoop QA commented on HDFS-6981:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666404/HDFS-6981.01.patch
against trunk revision 8f1a668.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7891//console
This message is automatically generated.
> DN upgrade with layout version change should not use trash
> --
>
> Key: HDFS-6981
> URL: https://issues.apache.org/jira/browse/HDFS-6981
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: James Thomas
>Assignee: Arpit Agarwal
> Attachments: HDFS-6981.01.patch
>
>
> Post HDFS-6800, we can encounter the following scenario:
> # We start with DN software version -55 and initiate a rolling upgrade to
> version -56
> # We delete some blocks, and they are moved to trash
> # We roll back to DN software version -55 using the -rollback flag – since we
> are running the old code (prior to this patch), we will restore the previous
> directory but will not delete the trash
> # We append to some of the blocks that were deleted in step 2
> # We then restart a DN that contains blocks that were appended to – since the
> trash still exists, it will be restored at this point, the appended-to blocks
> will be overwritten, and we will lose the appended data
> So I think we need to avoid writing anything to the trash directory if we
> have a previous directory.
> Thanks to [~james.thomas] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120891#comment-14120891 ] Arpit Agarwal commented on HDFS-6981: - Also thanks to James for finding this issue and proposing the fix. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas >Assignee: Arpit Agarwal > Attachments: HDFS-6981.01.patch > > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117787#comment-14117787 ] Arpit Agarwal commented on HDFS-6981: - The fix version can't be 2.6.0 since we have no timeline yet on getting the changes to branch-2. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117782#comment-14117782 ] James Thomas commented on HDFS-6981: [~arpitagarwal], thanks for filing this. Shouldn't the fix version be 2.6.0? This is a non-issue if we leave HDFS-6482 and HDFS-6800 in trunk, as I noted in https://issues.apache.org/jira/browse/HDFS-6800?focusedCommentId=14116260&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14116260. If we fix this JIRA, we can put HDFS-6482 and HDFS-6800 in branch-2. But trunk is currently fine as it is. > DN upgrade with layout version change should not use trash > -- > > Key: HDFS-6981 > URL: https://issues.apache.org/jira/browse/HDFS-6981 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0 >Reporter: James Thomas > > Post HDFS-6800, we can encounter the following scenario: > # We start with DN software version -55 and initiate a rolling upgrade to > version -56 > # We delete some blocks, and they are moved to trash > # We roll back to DN software version -55 using the -rollback flag – since we > are running the old code (prior to this patch), we will restore the previous > directory but will not delete the trash > # We append to some of the blocks that were deleted in step 2 > # We then restart a DN that contains blocks that were appended to – since the > trash still exists, it will be restored at this point, the appended-to blocks > will be overwritten, and we will lose the appended data > So I think we need to avoid writing anything to the trash directory if we > have a previous directory. > Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
