[jira] [Commented] (HDFS-14856) Add ability to import file ACLs from remote store
[
https://issues.apache.org/jira/browse/HDFS-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933600#comment-16933600
]
Ashvin commented on HDFS-14856:
---
cc: [~virajith] [~elgoiri]
> Add ability to import file ACLs from remote store
> -
>
> Key: HDFS-14856
> URL: https://issues.apache.org/jira/browse/HDFS-14856
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
>
> Provided storage (HDFS-9806) allows data on external storage systems to
> seamlessly appear as files on HDFS. However, in the implementation today, the
> external store scanner, {{FsTreeWalk,}} ignores any ACLs on the data. In a
> secure HDFS setup where external storage system and HDFS belong to the same
> security domain, uniform enforcement of the authorization policies may be
> desired. This task aims to extend the ability of the external store scanner
> to support this use case. When configured, the scanner should attempt to
> fetch ACLs and provide it to the consumer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14856) Add ability to import file ACLs from remote store
[
https://issues.apache.org/jira/browse/HDFS-14856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin reassigned HDFS-14856:
-
Assignee: Ashvin
> Add ability to import file ACLs from remote store
> -
>
> Key: HDFS-14856
> URL: https://issues.apache.org/jira/browse/HDFS-14856
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
>
> Provided storage (HDFS-9806) allows data on external storage systems to
> seamlessly appear as files on HDFS. However, in the implementation today, the
> external store scanner, {{FsTreeWalk,}} ignores any ACLs on the data. In a
> secure HDFS setup where external storage system and HDFS belong to the same
> security domain, uniform enforcement of the authorization policies may be
> desired. This task aims to extend the ability of the external store scanner
> to support this use case. When configured, the scanner should attempt to
> fetch ACLs and provide it to the consumer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14856) Add ability to import file ACLs from remote store
Ashvin created HDFS-14856:
-
Summary: Add ability to import file ACLs from remote store
Key: HDFS-14856
URL: https://issues.apache.org/jira/browse/HDFS-14856
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Ashvin
Provided storage (HDFS-9806) allows data on external storage systems to
seamlessly appear as files on HDFS. However, in the implementation today, the
external store scanner, {{FsTreeWalk,}} ignores any ACLs on the data. In a
secure HDFS setup where external storage system and HDFS belong to the same
security domain, uniform enforcement of the authorization policies may be
desired. This task aims to extend the ability of the external store scanner to
support this use case. When configured, the scanner should attempt to fetch
ACLs and provide it to the consumer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14805) Mounting external stores in HDFS on-the-fly
[
https://issues.apache.org/jira/browse/HDFS-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923770#comment-16923770
]
Ashvin commented on HDFS-14805:
---
Hi [~ehiggs],
Thanks for reviewing the design doc.
As of now there is no dependency on HDFS-12478. For most part the two proposed
commands are similar and this produces an opportunity to collaborate and reuse.
At the very least avoid conflicting contributions.
{noformat}
HDFS-14805:: hdfs dfsadmin -addMount []
HDFS-12478:: hdfs syncservice -create [-name ] -backupOnly
{noformat}
Some points to discuss:
# Should the "noun" be top level or under {{dfsadmin}}? given both the
operations are admin only operations, I think a subcommand under {{dfsadmin}}
fits well?
# I see you replaced {{attach}} with {{syncservice}}, how does mount sound to
you?
# I like the data movement "directionality" hint in the command you proposed.
Please let us know if you have any suggestions.
> Mounting external stores in HDFS on-the-fly
> ---
>
> Key: HDFS-14805
> URL: https://issues.apache.org/jira/browse/HDFS-14805
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Virajith Jalaparti
>Priority: Major
> Attachments: dynamic-mounts-in-hdfs.pdf
>
>
> Provided storage (HDFS-9806) allows HDFS to address data in external storage
> systems, including cloud stores. Data mounted in this manner, seamlessly,
> appears to be part of HDFS for applications/clients. The external data can
> also be cached by HDFS on local disks and SSDs, accelerating remote data
> reads (HDFS-13069).
> However, Provided storage was originally targeted at ephemeral HDFS
> deployments in the cloud (e.g., Azure HDInsight). Long running HDFS clusters
> are common in many other scenarios which can benefit from accessing data in
> remote stores. This JIRA targets such scenarios and aims to provide the
> ability to:
> (a) Dynamically mount external stores in a HDFS cluster while supporting high
> availability.
> (b) Mount multiple remote stores simultaneously.
> (c) Reduce deployment overheads and simplify usability of Provided storage.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834074#comment-16834074
]
Ashvin commented on HDFS-14390:
---
Thanks [~virajith]. Your suggestion to create util methods for common test code
setup makes sense to me. I have uploaded a new patch [^HDFS-14390.006.patch].
This patch moves kerberos test configuration to {{MiniDFSCluster}} on top of
the previous patch.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch,
> HDFS-14390.006.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.006.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch,
> HDFS-14390.006.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819585#comment-16819585
]
Ashvin commented on HDFS-14390:
---
Thanks for reviewing [~daryn].
I posted a new patch, [^HDFS-14390.005.patch]. This patch does not include
{{FSTreeWalk}} changes as they are not needed for fixing the issue.
Regarding test verification, the current test is not creating any files. This
is because in absence of the kerberos annotation,
{{TestSecureAliasMap.testSecureConnectionToAliasMap}} will fail to create
{{BlockAliasMap.Reader reader}} and the test will fail with the following
error. Accordingly I think the unit test is reliably verifying the connection.
I am inclining towards not complicating the unit test with alias map creation
details.
{{java.io.IOException: Unable to retrieve InMemoryAliasMap for block pool id
BP-1267604097-10.84.180.32-1555451080089}}{{ }}{{at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getAliasMap(InMemoryLevelDBAliasMapClient.java:173)}}{{at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getReader(InMemoryLevelDBAliasMapClient.java:180)}}{{at
org.apache.hadoop.hdfs.server.aliasmap.TestSecureAliasMap.testSecureConnectionToAliasMap(TestSecureAliasMap.java:198)}}
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.005.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch, HDFS-14390.005.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819200#comment-16819200
]
Ashvin commented on HDFS-14390:
---
Hi [~daryn] I wanted to follow up on this issue. I have uploaded a new patch,
[^HDFS-14390.004.patch]. As discussed earlier, the patch includes just the
changes required to enable secure connection to the {{AliasMap}} server and
fixes the image generation tool used for provided storage, see {{FSTreeWalk}}.
Auth related changes will be part of a new PR.
Summary: When authentication method is {{Kerberos}}, a client (DN/NN in this
case) invokes {{SaslRpcClient.getServerPrincipal}} to setup a secure
connection. If {{Provided storage}} is also enabled, the {{getServerPrincipal}}
method tries to find server principal for the {{AliasMap}} protocol. It was
missing earlier. This change, see {{AliasMapProtocolPB}}, provides the server
principal. No other components or paths will be impacted by this change.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.004.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch, HDFS-14390.004.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813856#comment-16813856
]
Ashvin commented on HDFS-14390:
---
Hi [~daryn] Thanks for looking at the changes.
When authentication method is {{Kerberos}}, a client (DN/NN in this case)
invokes {{SaslRpcClient.getServerPrincipal}} to setup a secure connection. If
{{Provided storage}} is also enabled, {{getServerPrincipal}} in turn tries to
find server principal for the {{AliasMap}} protocol. It was absent earlier.
This change, see {{AliasMapProtocolPB}}, provides the server principal.
I agree that the authz/acl related changes could be part of a different PR. The
{{FSTreeWalk}} changes are needed for the tool to establish a secure
connection. Does this change qualify a new PR?
I can post a new patch with the changes related to {{AliasMapProtocol}}
authentication in this PR. Please let me know if you have any other suggestions.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809281#comment-16809281
]
Ashvin commented on HDFS-14390:
---
[~elgoiri] [~virajith], thanks for the review.
Hi [~jlowe] [~crh], [~subru] and [~elgoiri] mentioned that your feedback would
be valuable. Could you please take a look at the changes. Thanks !
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.003.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804100#comment-16804100
]
Ashvin commented on HDFS-14390:
---
I uploaded a new patch [^HDFS-14390.002.patch]. As mentioned earlier, the
{{clientPrincipal}} is removed. It also addresses the {{javac}} warning. The
tests uses on {{MiniKdc}}. Based on javadoc and other tests using it, it seems
the recommended way to initialize it is in a {{static BeforeClass}} method.
Hence the new patch does not change the test setup.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.002.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803330#comment-16803330
]
Ashvin commented on HDFS-14390:
---
Thanks [~virajith] [~elgoiri] for reviewing the patch.
[~virajith], the {{KerberosInfo/clientPrincipal}} is used only if service level
authorization is enabled for the {{AliasMap}}. The {{clientPrincipal}} can be
removed when it is not configured and for the scope of this jira. Perhaps it
better to address service acl and authorization changes in a different patch?
[~elgoiri], reorganizing the test case and reusing security utils wherever
available makes sense. Will update the patch accordingly.
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: (was: HDFS-14390.001.patch)
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.001.patch
Status: Patch Available (was: Open)
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.001.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashvin updated HDFS-14390:
--
Attachment: HDFS-14390.001.patch
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802090#comment-16802090
]
Ashvin commented on HDFS-14390:
---
In a secure HDFS cluster, the DN and NN will fail to connect with the
{{AliasMap}} service. The following error messages can be seen in the logs.
2019-03-26 10:56:15,460 [Block report processor] WARN ipc.Client
(Client.java:run(760)) - Exception encountered while connecting to the server :
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[KERBEROS]
2019-03-26 10:56:15,461 [Block report processor] ERROR
impl.InMemoryLevelDBAliasMapClient
(InMemoryLevelDBAliasMapClient.java:getAliasMap(171)) - Exception in retrieving
block pool id {}
java.io.IOException: DestHost:destPort localhost:32445 , LocalHost:localPort
XXX. Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[KERBEROS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
…
at com.sun.proxy.$Proxy13.getBlockPoolId(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.InMemoryAliasMapProtocolClientSideTranslatorPB.getBlockPoolId(InMemoryAliasMapProtocolClientSideTranslatorPB.java:219)
at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getAliasMap(InMemoryLevelDBAliasMapClient.java:165)
at
org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.InMemoryLevelDBAliasMapClient.getReader(InMemoryLevelDBAliasMapClient.java:181)
at
org.apache.hadoop.hdfs.server.blockmanagement.ProvidedStorageMap.processProvidedStorageReport(ProvidedStorageMap.java:156)
at
org.apache.hadoop.hdfs.server.blockmanagement.ProvidedStorageMap.getStorage(ProvidedStorageMap.java:139)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:2536)
…
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:765)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:728)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:822)
…
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Priority: Major
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
Ashvin created HDFS-14390:
-
Summary: Provide kerberos support for AliasMap service used by
Provided storage
Key: HDFS-14390
URL: https://issues.apache.org/jira/browse/HDFS-14390
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ashvin
With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
external storage systems. This feature is not supported in a secure HDFS
cluster. The {{AliasMap}} service does not support kerberos, and as a result
the cluster nodes will fail to communicate with it. This JIRA is to enable
kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
