[jira] [Updated] (HDFS-7555) Remove the support of unmanaged connectors in HttpServer2
[ https://issues.apache.org/jira/browse/HDFS-7555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7555: - Attachment: HDFS-7555.003.patch > Remove the support of unmanaged connectors in HttpServer2 > - > > Key: HDFS-7555 > URL: https://issues.apache.org/jira/browse/HDFS-7555 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-7555.001.patch, HDFS-7555.002.patch, > HDFS-7555.003.patch > > > After HDFS-7279 there is no need to support unmanaged connectors in > HttpServer2. This jira proposes to remove the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7555) Remove the support of unmanaged connectors in HttpServer2
[
https://issues.apache.org/jira/browse/HDFS-7555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254600#comment-14254600
]
Hadoop QA commented on HDFS-7555:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688503/HDFS-7555.003.patch
against trunk revision c1f857b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
org.apache.hadoop.ha.TestZKFailoverControllerStress
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9101//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9101//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9101//console
This message is automatically generated.
> Remove the support of unmanaged connectors in HttpServer2
> -
>
> Key: HDFS-7555
> URL: https://issues.apache.org/jira/browse/HDFS-7555
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-7555.001.patch, HDFS-7555.002.patch,
> HDFS-7555.003.patch
>
>
> After HDFS-7279 there is no need to support unmanaged connectors in
> HttpServer2. This jira proposes to remove the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254662#comment-14254662
]
Hudson commented on HDFS-7552:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #47 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/47/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254666#comment-14254666
]
Hudson commented on HDFS-7443:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #47 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/47/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254676#comment-14254676
]
Hudson commented on HDFS-7552:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #781 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/781/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254680#comment-14254680
]
Hudson commented on HDFS-7443:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #781 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/781/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254743#comment-14254743
]
Hudson commented on HDFS-7443:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1979 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1979/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254739#comment-14254739
]
Hudson commented on HDFS-7552:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1979 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1979/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254757#comment-14254757
]
Hudson commented on HDFS-7443:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #44 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/44/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254753#comment-14254753
]
Hudson commented on HDFS-7552:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #44 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/44/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254788#comment-14254788
]
Hudson commented on HDFS-7552:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #48 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/48/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254792#comment-14254792
]
Hudson commented on HDFS-7443:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #48 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/48/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7551) Fix the new findbugs warning from TransferFsImage
[
https://issues.apache.org/jira/browse/HDFS-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254801#comment-14254801
]
Akira AJISAKA commented on HDFS-7551:
-
+1, these test failures are unrelated to the patch.
> Fix the new findbugs warning from TransferFsImage
> -
>
> Key: HDFS-7551
> URL: https://issues.apache.org/jira/browse/HDFS-7551
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Minor
> Attachments: HDFS-7551-001.txt, HDFS-7551-002.txt
>
>
> There is a findbug warning in
> https://builds.apache.org/job/PreCommit-HDFS-Build/9080//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
> ,
> {code}
> Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
> In class org.apache.hadoop.hdfs.server.namenode.TransferFsImage
> In method
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List)
> Called method java.io.File.delete()
> At TransferFsImage.java:[line 577]
> {code}
> seems to me it came from https://issues.apache.org/jira/browse/HDFS-7373 's
> change
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7551) Fix the new findbugs warning from TransferFsImage
[
https://issues.apache.org/jira/browse/HDFS-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-7551:
Hadoop Flags: Reviewed
> Fix the new findbugs warning from TransferFsImage
> -
>
> Key: HDFS-7551
> URL: https://issues.apache.org/jira/browse/HDFS-7551
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Minor
> Attachments: HDFS-7551-001.txt, HDFS-7551-002.txt
>
>
> There is a findbug warning in
> https://builds.apache.org/job/PreCommit-HDFS-Build/9080//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
> ,
> {code}
> Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
> In class org.apache.hadoop.hdfs.server.namenode.TransferFsImage
> In method
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List)
> Called method java.io.File.delete()
> At TransferFsImage.java:[line 577]
> {code}
> seems to me it came from https://issues.apache.org/jira/browse/HDFS-7373 's
> change
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7551) Fix the new findbugs warning from TransferFsImage
[
https://issues.apache.org/jira/browse/HDFS-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-7551:
Target Version/s: 2.7.0
> Fix the new findbugs warning from TransferFsImage
> -
>
> Key: HDFS-7551
> URL: https://issues.apache.org/jira/browse/HDFS-7551
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Minor
> Attachments: HDFS-7551-001.txt, HDFS-7551-002.txt
>
>
> There is a findbug warning in
> https://builds.apache.org/job/PreCommit-HDFS-Build/9080//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
> ,
> {code}
> Bug type RV_RETURN_VALUE_IGNORED_BAD_PRACTICE
> In class org.apache.hadoop.hdfs.server.namenode.TransferFsImage
> In method
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.deleteTmpFiles(List)
> Called method java.io.File.delete()
> At TransferFsImage.java:[line 577]
> {code}
> seems to me it came from https://issues.apache.org/jira/browse/HDFS-7373 's
> change
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7373) Clean up temporary files after fsimage transfer failures
[ https://issues.apache.org/jira/browse/HDFS-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7373: Fix Version/s: 2.7.0 > Clean up temporary files after fsimage transfer failures > > > Key: HDFS-7373 > URL: https://issues.apache.org/jira/browse/HDFS-7373 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 2.7.0 > > Attachments: HDFS-7373.patch > > > When a fsimage (e.g. checkpoint) transfer fails, a temporary file is left in > each storage directory. If the size of name space is large, these files can > take up quite a bit of space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7552) change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
[
https://issues.apache.org/jira/browse/HDFS-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254805#comment-14254805
]
Hudson commented on HDFS-7552:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1998 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1998/])
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) (cmccabe:
rev a4876c130f1627e59ef055e586640d1933fc49af)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
> --
>
> Key: HDFS-7552
> URL: https://issues.apache.org/jira/browse/HDFS-7552
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 2.7.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Fix For: 2.7.0
>
> Attachments: HDFS-7552-001.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9088//testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureToleration/testValidVolumesAtStartup/
> Per my understanding, it was due to:
> commit 3b173d95171d01ab55042b1162569d1cf14a8d43
> Author: Colin Patrick Mccabe
> Date: Wed Dec 17 16:41:59 2014 -0800
> HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via
> Colin P. McCabe)
> - volatile List volumes = null;
> + private final AtomicReference volumes =
> + new AtomicReference<>(new FsVolumeImpl[0]);
> so the old case will complain at here:
> {code}
> assertTrue("The DN should have started with this directory",
> si.contains(dataDir1Actual.getPath()));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7443) Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume
[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254809#comment-14254809
]
Hudson commented on HDFS-7443:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1998 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1998/])
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block
files are present in the same volume (cmccabe) (cmccabe: rev
8fa265a290792ff42635ff9b42416c634f88bdf3)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-24-datanode-dir.tgz
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> --
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: HDFS-7443.001.patch, HDFS-7443.002.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254816#comment-14254816 ] Akira AJISAKA commented on HDFS-7182: - +1 > JMX metrics aren't accessible when NN is busy > - > > Key: HDFS-7182 > URL: https://issues.apache.org/jira/browse/HDFS-7182 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HDFS-7182-2.patch, HDFS-7182.patch > > > HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple > new metrics have been added. It turns out "RollingUpgradeStatus" requires > FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7182: Target Version/s: 2.7.0 Hadoop Flags: Reviewed > JMX metrics aren't accessible when NN is busy > - > > Key: HDFS-7182 > URL: https://issues.apache.org/jira/browse/HDFS-7182 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HDFS-7182-2.patch, HDFS-7182.patch > > > HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple > new metrics have been added. It turns out "RollingUpgradeStatus" requires > FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7560) ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover
Vinayakumar B created HDFS-7560:
---
Summary: ACLs removed by removeDefaultAcl() will be back after
NameNode restart/failover
Key: HDFS-7560
URL: https://issues.apache.org/jira/browse/HDFS-7560
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.1
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical
Default ACLs removed using {{removeDefaultAcl()}} will come back after Namenode
restart/switch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7560) ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover
[
https://issues.apache.org/jira/browse/HDFS-7560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-7560:
Attachment: HDFS-7560-001.patch
Attached the patch for the same. Please review.
> ACLs removed by removeDefaultAcl() will be back after NameNode
> restart/failover
> ---
>
> Key: HDFS-7560
> URL: https://issues.apache.org/jira/browse/HDFS-7560
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.1
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-7560-001.patch
>
>
> Default ACLs removed using {{removeDefaultAcl()}} will come back after
> Namenode restart/switch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7560) ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover
[
https://issues.apache.org/jira/browse/HDFS-7560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-7560:
Status: Patch Available (was: Open)
> ACLs removed by removeDefaultAcl() will be back after NameNode
> restart/failover
> ---
>
> Key: HDFS-7560
> URL: https://issues.apache.org/jira/browse/HDFS-7560
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.1
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-7560-001.patch
>
>
> Default ACLs removed using {{removeDefaultAcl()}} will come back after
> Namenode restart/switch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7456) De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode
[
https://issues.apache.org/jira/browse/HDFS-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-7456:
Attachment: HDFS-7456-007.patch
Addressed above comments.
Thanks Again Chris for the suggestion of verifying the counts. While testing
that found HDFS-7560, seems a serious bug.
Though I could verify counts for AClFeatures of all tests, but I intentionally
reinited the cluster in {{testDeduplication}} in {{FSAclBaseTest}} to avoid
failures due to restart of the cluster in some other tests ( ex: HDFS-7560) .
Please review.
> De-duplicate AclFeature instances with same AclEntries do reduce memory
> footprint of NameNode
> -
>
> Key: HDFS-7456
> URL: https://issues.apache.org/jira/browse/HDFS-7456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-7456-001.patch, HDFS-7456-002.patch,
> HDFS-7456-003.patch, HDFS-7456-004.patch, HDFS-7456-005.patch,
> HDFS-7456-006.patch, HDFS-7456-007.patch
>
>
> As discussed in HDFS-7454
> [here|https://issues.apache.org/jira/browse/HDFS-7454?focusedCommentId=14229454&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14229454],
> de-duplication of {{AclFeature}} helps in reducing the memory footprint of
> the namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7014) Implement input streams and file system functionality
[ https://issues.apache.org/jira/browse/HDFS-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254951#comment-14254951 ] Suresh Srinivas commented on HDFS-7014: --- [~cmccabe] and [~wangzw], I am going to post a similar comment to the umbrella jira. This is a large c++ code base. Did we decide on coding standards, which is very critical to ensure consistency and correctness. What is the plan for testing. We generally have not allowed code without tests into the code base. We have been lax about this. It would be idea to get the tests along with the code during commits. > Implement input streams and file system functionality > - > > Key: HDFS-7014 > URL: https://issues.apache.org/jira/browse/HDFS-7014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Fix For: HDFS-6994 > > Attachments: 0001-HDFS-7014-001.patch, HDFS-7014-pnative.002.patch, > HDFS-7014-pnative.003.patch, HDFS-7014-pnative.004.patch, HDFS-7014.patch > > > Implement Client - Namenode RPC protocol and support Namenode HA. > Implement Client - Datanode RPC protocol. > Implement some basic server side class such as ExtendedBlock and LocatedBlock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254953#comment-14254953 ] Suresh Srinivas commented on HDFS-6994: --- Folks, I have not caught up with all the comments. I wanted to know how we want to handle the following things: - Coding guidelines - we should adopt one. We could perhaps start with [google cpp guide|https://google-styleguide.googlecode.com/svn/trunk/cppguide.html] - As we have started implementing large code base in c/cpp, we need unit tests. We should not be committing code without unit tests. We could adopt [google test|https://code.google.com/p/googletest/] Without these two prerequisites, we will not be able to have consistent coding, ability to effectively collaborate within the community and no way to unit test and ensure correctness. > libhdfs3 - A native C/C++ HDFS client > - > > Key: HDFS-6994 > URL: https://issues.apache.org/jira/browse/HDFS-6994 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Zhanwei Wang >Assignee: Zhanwei Wang > Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch > > > Hi All > I just got the permission to open source libhdfs3, which is a native C/C++ > HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol. > libhdfs3 provide the libhdfs style C interface and a C++ interface. Support > both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos > authentication. > libhdfs3 is currently used by HAWQ of Pivotal > I'd like to integrate libhdfs3 into HDFS source code to benefit others. > You can find libhdfs3 code from github > https://github.com/PivotalRD/libhdfs3 > http://pivotalrd.github.io/libhdfs3/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7560) ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover
[
https://issues.apache.org/jira/browse/HDFS-7560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254982#comment-14254982
]
Hadoop QA commented on HDFS-7560:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688551/HDFS-7560-001.patch
against trunk revision 76b0370.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9102//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9102//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9102//console
This message is automatically generated.
> ACLs removed by removeDefaultAcl() will be back after NameNode
> restart/failover
> ---
>
> Key: HDFS-7560
> URL: https://issues.apache.org/jira/browse/HDFS-7560
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.5.1
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-7560-001.patch
>
>
> Default ACLs removed using {{removeDefaultAcl()}} will come back after
> Namenode restart/switch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7456) De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode
[
https://issues.apache.org/jira/browse/HDFS-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254986#comment-14254986
]
Hadoop QA commented on HDFS-7456:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688553/HDFS-7456-007.patch
against trunk revision 24ee9e3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
org.apache.hadoop.fs.TestSymlinkHdfsFileContext
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9103//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9103//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9103//console
This message is automatically generated.
> De-duplicate AclFeature instances with same AclEntries do reduce memory
> footprint of NameNode
> -
>
> Key: HDFS-7456
> URL: https://issues.apache.org/jira/browse/HDFS-7456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-7456-001.patch, HDFS-7456-002.patch,
> HDFS-7456-003.patch, HDFS-7456-004.patch, HDFS-7456-005.patch,
> HDFS-7456-006.patch, HDFS-7456-007.patch
>
>
> As discussed in HDFS-7454
> [here|https://issues.apache.org/jira/browse/HDFS-7454?focusedCommentId=14229454&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14229454],
> de-duplication of {{AclFeature}} helps in reducing the memory footprint of
> the namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode
[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254995#comment-14254995
]
Tsuyoshi OZAWA commented on HDFS-6833:
--
[~szetszwo] [~cnauroth] do you mind taking a look?
> DirectoryScanner should not register a deleting block with memory of DataNode
> -
>
> Key: HDFS-6833
> URL: https://issues.apache.org/jira/browse/HDFS-6833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.5.0, 2.5.1
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Critical
> Attachments: HDFS-6833-10.patch, HDFS-6833-11.patch,
> HDFS-6833-12.patch, HDFS-6833-13.patch, HDFS-6833-6-2.patch,
> HDFS-6833-6-3.patch, HDFS-6833-6.patch, HDFS-6833-7-2.patch,
> HDFS-6833-7.patch, HDFS-6833.8.patch, HDFS-6833.9.patch, HDFS-6833.patch,
> HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch
>
>
> When a block is deleted in DataNode, the following messages are usually
> output.
> {code}
> 2014-08-07 17:53:11,606 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:11,617 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> However, DirectoryScanner may be executed when DataNode deletes the block in
> the current implementation. And the following messsages are output.
> {code}
> 2014-08-07 17:53:30,519 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:31,426 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
> files:0, missing block files:0, missing blocks in memory:1, mismatched
> blocks:0
> 2014-08-07 17:53:31,426 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
> missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
> getNumBytes() = 21230663
> getBytesOnDisk() = 21230663
> getVisibleLength()= 21230663
> getVolume() = /hadoop/data1/dfs/data/current
> getBlockFile()=
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> unlinked =false
> 2014-08-07 17:53:31,531 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> Deleting block information is registered in DataNode's memory.
> And when DataNode sends a block report, NameNode receives wrong block
> information.
> For example, when we execute recommission or change the number of
> replication, NameNode may delete the right block as "ExcessReplicate" by this
> problem.
> And "Under-Replicated Blocks" and "Missing Blocks" occur.
> When DataNode run DirectoryScanner, DataNode should not register a deleting
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[
https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255032#comment-14255032
]
Kai Zheng commented on HDFS-6133:
-
Hi Yunjiong,
The patch looks great. Minor comments:
1. Regarding the following codes, are you only putting the first DN as favored
node or all the DNs.
{code}
+ InetSocketAddress[] favoredNodes = new InetSocketAddress[numOfDatanodes];
+ for (int i = 0; i < favoredNodes.length; i++) {
+favoredNodes[i] = cluster.getDataNodes().get(0).getXferAddress();
+ }
{code}
2. In DistributedFileSystem.java, there is a comment as below. It would be
better to be updated as well.
{code}
/**
* Same as
* {@link #create(Path, FsPermission, boolean, int, short, long,
* Progressable)} with the addition of favoredNodes that is a hint to
* where the namenode should place the file blocks.
* The favored nodes hint is not persisted in HDFS. Hence it may be honored
* at the creation time only. HDFS could move the blocks during balancing or
* replication, to move the blocks from favored nodes. A value of null means
* no favored nodes for this create
*/
{code}
3. By the way, a question: do we need any handling for favored nodes and the
pin way considering storage type/policy/Mover?
Thanks.
> Make Balancer support exclude specified path
>
>
> Key: HDFS-6133
> URL: https://issues.apache.org/jira/browse/HDFS-6133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Reporter: zhaoyunjiong
>Assignee: zhaoyunjiong
> Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch,
> HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133.patch
>
>
> Currently, run Balancer will destroying Regionserver's data locality.
> If getBlocks could exclude blocks belongs to files which have specific path
> prefix, like "/hbase", then we can run Balancer without destroying
> Regionserver's data locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7456) De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode
[
https://issues.apache.org/jira/browse/HDFS-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-7456:
Attachment: HDFS-7456-008.patch
Corrected the snapshot test failure.
> De-duplicate AclFeature instances with same AclEntries do reduce memory
> footprint of NameNode
> -
>
> Key: HDFS-7456
> URL: https://issues.apache.org/jira/browse/HDFS-7456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-7456-001.patch, HDFS-7456-002.patch,
> HDFS-7456-003.patch, HDFS-7456-004.patch, HDFS-7456-005.patch,
> HDFS-7456-006.patch, HDFS-7456-007.patch, HDFS-7456-008.patch
>
>
> As discussed in HDFS-7454
> [here|https://issues.apache.org/jira/browse/HDFS-7454?focusedCommentId=14229454&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14229454],
> de-duplication of {{AclFeature}} helps in reducing the memory footprint of
> the namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
