snvijaya commented on a change in pull request #1785: [Doc] Update abfs.md to include additional configs URL: https://github.com/apache/hadoop/pull/1785#discussion_r368377716
########## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ########## @@ -857,6 +857,46 @@ signon page for humans, even though it is a machine calling. 1. The URL is wrong —it is pointing at a web page unrelated to OAuth2.0 1. There's a proxy server in the way trying to return helpful instructions. +### `java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/user1/.staging is not as expected. It is owned by <principal_id>. The directory must be owned by the submitter user1 or user1` + +When using [Azure Managed Identities](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview), the files/directories in ADLS Gen2 are by default owned by the service principal object id i.e. principal ID & submitting jobs as the local OS user 'user1' results in the above exception. + +The following configurations have to be added to core-site.xml to resolve this issue + +```xml +<property> + <name>fs.azure.identity.transformer.service.principal.id</name> + <value>service principal object id</value> + <description> + An Azure Active Directory object ID (oid) used as the replacement for names contained + in the list specified by “fs.azure.identity.transformer.service.principal.substitution.list”. + Notice that instead of setting oid, you can also set $superuser here. + </description> +</property> +<property> + <name>fs.azure.identity.transformer.service.principal.substitution.list</name> + <value>user1</value> + <description> + A comma separated list of names to be replaced with the service principal ID specified by + “fs.azure.identity.transformer.service.principal.id”. This substitution occurs + when setOwner, setAcl, modifyAclEntries, or removeAclEntries are invoked with identities + contained in the substitution list. Notice that when in non-secure cluster, asterisk symbol * + can be used to match all user/group. + </description> +</property> +<property> + <name>fs.azure.use.upn</name> Review comment: This config is not required to be set as it does not contribute to the conversion of the AAD OID to local user. Can be removed from this PR. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
