[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840698#comment-16840698 ] Hudson commented on HDFS-14390: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16555 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16555/]) HDFS-14390. Provide kerberos support for AliasMap service used by (virajith: rev 77170e70d16e309121ca7730974617c05e66d063) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/aliasmap/InMemoryLevelDBAliasMapServer.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/aliasmap/TestSecureAliasMap.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/AliasMapProtocolPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/impl/InMemoryLevelDBAliasMapClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ProvidedStorageMap.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch, > HDFS-14390.006.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840685#comment-16840685 ] Virajith Jalaparti commented on HDFS-14390: --- Committed [^HDFS-14390.006.patch] to trunk. Thanks [~ashvin] for the patch and [~elgoiri] [~daryn] for the review. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch, > HDFS-14390.006.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835229#comment-16835229 ] Íñigo Goiri commented on HDFS-14390: +1 from my side. On Tue, May 7, 2019, 07:00 Virajith Jalaparti (JIRA) > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch, > HDFS-14390.006.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834949#comment-16834949 ] Virajith Jalaparti commented on HDFS-14390: --- Thanks for making the changes [~ashvin]. [^HDFS-14390.006.patch] looks good to me. [~elgoiri], [~daryn] - if you don't have any more comments on [^HDFS-14390.006.patch], I'll commit it. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch, > HDFS-14390.006.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834180#comment-16834180 ] Hadoop QA commented on HDFS-14390: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 5s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}169m 41s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController | | | hadoop.hdfs.web.TestWebHdfsFileSystemContract | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967960/HDFS-14390.006.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 824ac84dd005 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 12b7059 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26754/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26754/testReport/ | | Max. process+thread count | 3015 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26754/console | | Powered by |
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834074#comment-16834074 ] Ashvin commented on HDFS-14390: --- Thanks [~virajith]. Your suggestion to create util methods for common test code setup makes sense to me. I have uploaded a new patch [^HDFS-14390.006.patch]. This patch moves kerberos test configuration to {{MiniDFSCluster}} on top of the previous patch. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch, > HDFS-14390.006.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832720#comment-16832720 ] Virajith Jalaparti commented on HDFS-14390: --- Thanks for working on this [~ashvin]. The code in {{TestSecureAliasMap#init}} seems duplicated from other tests. Can we move this code to {{MiniDFSCluster}} (or related) so that future tests (and existing tests) can use it and not have to duplicate it? Eventually, existing tests should be refactored to use this utility instead of setting up their own configs. Other than that +1 on [^HDFS-14390.005.patch] . [~daryn] do you have any other comments? > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819646#comment-16819646 ] Hadoop QA commented on HDFS-14390: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 28s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}141m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.web.TestWebHdfsTimeouts | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12966157/HDFS-14390.005.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 11820fc413dc 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e543c3b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26651/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26651/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results |
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819585#comment-16819585 ] Ashvin commented on HDFS-14390: --- Thanks for reviewing [~daryn]. I posted a new patch, [^HDFS-14390.005.patch]. This patch does not include {{FSTreeWalk}} changes as they are not needed for fixing the issue. Regarding test verification, the current test is not creating any files. This is because in absence of the kerberos annotation, {{TestSecureAliasMap.testSecureConnectionToAliasMap}} will fail to create {{BlockAliasMap.Reader reader}} and the test will fail with the following error. Accordingly I think the unit test is reliably verifying the connection. I am inclining towards not complicating the unit test with alias map creation details. {{java.io.IOException: Unable to retrieve InMemoryAliasMap for block pool id BP-1267604097-10.84.180.32-1555451080089}}{{ }}{{at org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getAliasMap(InMemoryLevelDBAliasMapClient.java:173)}}{{at org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getReader(InMemoryLevelDBAliasMapClient.java:180)}}{{at org.apache.hadoop.hdfs.server.aliasmap.TestSecureAliasMap.testSecureConnectionToAliasMap(TestSecureAliasMap.java:198)}} > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819252#comment-16819252 ] Daryn Sharp commented on HDFS-14390: Ok, so the kerberos annotation is effectively the only change. You should make a call via the fs instance, ex. getServerDefaults, to ensure that it can actually connect and authenticate. The mini-cluster has probably already used the fs but I wouldn't rely on it. I don't think the {{FSTreeWalk}} change is a good idea. Is it actually needed? The login user is the current user unless an explicit doAs another identity has been done in which it was probably done for good reason. Why add a latent surprise that second-guesses the caller and reverts back to the login user? > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819200#comment-16819200 ] Ashvin commented on HDFS-14390: --- Hi [~daryn] I wanted to follow up on this issue. I have uploaded a new patch, [^HDFS-14390.004.patch]. As discussed earlier, the patch includes just the changes required to enable secure connection to the {{AliasMap}} server and fixes the image generation tool used for provided storage, see {{FSTreeWalk}}. Auth related changes will be part of a new PR. Summary: When authentication method is {{Kerberos}}, a client (DN/NN in this case) invokes {{SaslRpcClient.getServerPrincipal}} to setup a secure connection. If {{Provided storage}} is also enabled, the {{getServerPrincipal}} method tries to find server principal for the {{AliasMap}} protocol. It was missing earlier. This change, see {{AliasMapProtocolPB}}, provides the server principal. No other components or paths will be impacted by this change. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch, HDFS-14390.004.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818849#comment-16818849 ] Hadoop QA commented on HDFS-14390: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 15s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}106m 51s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s{color} | {color:green} hadoop-fs2img in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 55s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}208m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12966036/HDFS-14390.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux cc17362a9d01 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a5ceed2 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26643/testReport/ | | Max. process+thread count | 2885 (vs. ulimit of 1) | | modules | C:
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813856#comment-16813856 ] Ashvin commented on HDFS-14390: --- Hi [~daryn] Thanks for looking at the changes. When authentication method is {{Kerberos}}, a client (DN/NN in this case) invokes {{SaslRpcClient.getServerPrincipal}} to setup a secure connection. If {{Provided storage}} is also enabled, {{getServerPrincipal}} in turn tries to find server principal for the {{AliasMap}} protocol. It was absent earlier. This change, see {{AliasMapProtocolPB}}, provides the server principal. I agree that the authz/acl related changes could be part of a different PR. The {{FSTreeWalk}} changes are needed for the tool to establish a secure connection. Does this change qualify a new PR? I can post a new patch with the changes related to {{AliasMapProtocol}} authentication in this PR. Please let me know if you have any other suggestions. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813586#comment-16813586 ] Daryn Sharp commented on HDFS-14390: Jason asked me to take a look. Which change actually fixed the unavailability of the kerberos credentials (ugi context) in the block report processing thread? All I see are formerly cited unrelated acl/authz changes that really should be a separate jira. The only ugi context change I see is in {{FSTreeWalk}} which is part of a tool. How does that affect the block report processing thread? (Aside: I'm rather astounded that block report processing is apparently calling out to a DN RPC service?) > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809281#comment-16809281 ] Ashvin commented on HDFS-14390: --- [~elgoiri] [~virajith], thanks for the review. Hi [~jlowe] [~crh], [~subru] and [~elgoiri] mentioned that your feedback would be valuable. Could you please take a look at the changes. Thanks ! > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807955#comment-16807955 ] Íñigo Goiri commented on HDFS-14390: +1 on [^HDFS-14390.003.patch]. I'd like [~virajith] to take a look too. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch, > HDFS-14390.003.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807229#comment-16807229 ] Hadoop QA commented on HDFS-14390: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 22s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 0s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s{color} | {color:green} hadoop-fs2img in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 59s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}245m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.server.datanode.TestBPOfferService | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12964467/HDFS-14390.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2558aaa94ea0 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806074#comment-16806074 ] Íñigo Goiri commented on HDFS-14390: Thanks for [^HDFS-14390.002.patch]. Just a minor comment, can we avoid the empty line change at the end of FSTreeWalk? Other than that LGTM. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804331#comment-16804331 ] Hadoop QA commented on HDFS-14390: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 3s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 15s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s{color} | {color:green} hadoop-fs2img in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}189m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestMaintenanceState | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12964055/HDFS-14390.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 59d88d55e330 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 49b02d4 | |
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804100#comment-16804100 ] Ashvin commented on HDFS-14390: --- I uploaded a new patch [^HDFS-14390.002.patch]. As mentioned earlier, the {{clientPrincipal}} is removed. It also addresses the {{javac}} warning. The tests uses on {{MiniKdc}}. Based on javadoc and other tests using it, it seems the recommended way to initialize it is in a {{static BeforeClass}} method. Hence the new patch does not change the test setup. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803330#comment-16803330 ] Ashvin commented on HDFS-14390: --- Thanks [~virajith] [~elgoiri] for reviewing the patch. [~virajith], the {{KerberosInfo/clientPrincipal}} is used only if service level authorization is enabled for the {{AliasMap}}. The {{clientPrincipal}} can be removed when it is not configured and for the scope of this jira. Perhaps it better to address service acl and authorization changes in a different patch? [~elgoiri], reorganizing the test case and reusing security utils wherever available makes sense. Will update the patch accordingly. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802553#comment-16802553 ] Íñigo Goiri commented on HDFS-14390: Thanks [~ashvin] for the patch. The unit test failure is unrelated and present in many other builds. The added test seems to work as expected; https://builds.apache.org/job/PreCommit-HDFS-Build/26521/testReport/org.apache.hadoop.hdfs.server.aliasmap/TestSecureAliasMap/ A couple minor comments: * Let's fix the javac warning. * I probably would do the setup of the test as @Before and @After (without statics). * Can we use some of the utilities already present for security? For example SecurityUtil or so. Regarding the comment from [~virajith], I think that's a reasonable assumption. Maybe we can check for that in the unit test for it to provide a reference? > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Assignee: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802501#comment-16802501 ] Hadoop QA commented on HDFS-14390: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 59s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 24s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 24s{color} | {color:red} root generated 1 new + 1482 unchanged - 0 fixed = 1483 total (was 1482) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 28s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 14s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s{color} | {color:green} hadoop-fs2img in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}188m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14390 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12963824/HDFS-14390.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fd2e6344f4fc 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk /
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802450#comment-16802450 ] Virajith Jalaparti commented on HDFS-14390: --- Thanks for reporting and working on this [~ashvin]. The patch looks good to me. My only concern is that for this to work the namenode and datanode principles have to be the same if {{InMemoryAliasMapClient.java}} is used by both. > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Priority: Major > Attachments: HDFS-14390.001.patch > > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[ https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802090#comment-16802090 ] Ashvin commented on HDFS-14390: --- In a secure HDFS cluster, the DN and NN will fail to connect with the {{AliasMap}} service. The following error messages can be seen in the logs. 2019-03-26 10:56:15,460 [Block report processor] WARN ipc.Client (Client.java:run(760)) - Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS] 2019-03-26 10:56:15,461 [Block report processor] ERROR impl.InMemoryLevelDBAliasMapClient (InMemoryLevelDBAliasMapClient.java:getAliasMap(171)) - Exception in retrieving block pool id {} java.io.IOException: DestHost:destPort localhost:32445 , LocalHost:localPort XXX. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) … at com.sun.proxy.$Proxy13.getBlockPoolId(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.InMemoryAliasMapProtocolClientSideTranslatorPB.getBlockPoolId(InMemoryAliasMapProtocolClientSideTranslatorPB.java:219) at org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getAliasMap(InMemoryLevelDBAliasMapClient.java:165) at org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getReader(InMemoryLevelDBAliasMapClient.java:181) at org.apache.hadoop.hdfs.server.blockmanagement.ProvidedStorageMap.processProvidedStorageReport(ProvidedStorageMap.java:156) at org.apache.hadoop.hdfs.server.blockmanagement.ProvidedStorageMap.getStorage(ProvidedStorageMap.java:139) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:2536) … Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:765) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:728) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822) … > Provide kerberos support for AliasMap service used by Provided storage > -- > > Key: HDFS-14390 > URL: https://issues.apache.org/jira/browse/HDFS-14390 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ashvin >Priority: Major > > With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in > external storage systems. This feature is not supported in a secure HDFS > cluster. The {{AliasMap}} service does not support kerberos, and as a result > the cluster nodes will fail to communicate with it. This JIRA is to enable > kerberos support for the {{AliasMap}} service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org