[
https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776930#comment-16776930
]
Steve Loughran commented on HADOOP-16068:
-----------------------------------------
log on oauth returns HTML for next patch. Doesn't fix problem, but helps
{code}
bin/hadoop fs -ls abfs://contai...@abfswales1.dfs.core.windows.net/
14:22:01
2019-02-25 14:23:07,975 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:initialize(101)) - Initializing AzureBlobFileSystem
for abfs://contai...@abfswales1.dfs.core.windows.net/
2019-02-25 14:23:08,182 [main] DEBUG services.AbfsClientThrottlingIntercept
(AbfsClientThrottlingIntercept.java:initializeSingleton(62)) - Client-side
throttling is enabled for the ABFS file system.
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:getFileStatus(434)) -
AzureBlobFileSystem.getFileStatus path:
abfs://contai...@abfswales1.dfs.core.windows.net/
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:performAbfsAuthCheck(1101)) - ABFS authorizer is not
initialized. No authorization check will be performed.
2019-02-25 14:23:08,204 [main] DEBUG azurebfs.AzureBlobFileSystemStore
(AzureBlobFileSystemStore.java:getIsNamespaceEnabled(200)) - Get root ACL status
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(AccessTokenProvider.java:isTokenAboutToExpire(77)) - AADToken: no token.
Returning expiring=true
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(AccessTokenProvider.java:getToken(48)) - AAD Token is missing or expired:
Calling refresh-token from abstract base class
2019-02-25 14:23:08,272 [main] DEBUG oauth2.AccessTokenProvider
(ClientCredsTokenProvider.java:refreshToken(57)) - AADToken: refreshing
client-credential based token
2019-02-25 14:23:08,274 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenUsingClientCreds(94)) - AADToken: starting
to fetch token using client creds for client ID
40c8b4e5-865a-4297-bbbd-195df2a8a806
2019-02-25 14:23:08,274 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenSingleCall(297)) - Requesting an OAuth token
by POST to
https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize
2019-02-25 14:23:08,275 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Request Headers
2019-02-25 14:23:08,275 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Connection=close
2019-02-25 14:23:08,872 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenSingleCall(319)) - Response 200
2019-02-25 14:23:08,872 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Response Headers
2019-02-25 14:23:08,873 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - HTTP Response=HTTP/1.1 200 OK
2019-02-25 14:23:08,873 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Server=Microsoft-IIS/10.0
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - X-Content-Type-Options=nosniff
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Connection=close
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Pragma=no-cache
2019-02-25 14:23:08,874 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - P3P=CP="DSP CUR OTPi IND OTRi
ONL FIN"
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Date=Mon, 25 Feb 2019 14:23:08
GMT
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - X-Frame-Options=DENY
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
Strict-Transport-Security=max-age=31536000; includeSubDomains
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Cache-Control=no-cache,
no-store
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
Set-Cookie=stsservicecookie=ests; path=/; secure;
HttpOnly;x-ms-gateway-slice=prod; path=/; secure;
HttpOnly;esctx=AQABAAAAAACEfexXxjamQb3OeGQ4GugvJnqCleb5fTHXUNtG2e2vTfvF4cV_3pPI9WICNEPT-85W2F8bAZgHI_L1btZuYPY5pru5civf8jCRgzmUuU5mfcyeeYApPXDdetus1cHPtPEe7cNjOewm_S0dVIvGvA3pS87ggExUYAXwnRzRj55Z1T7ftN444Vg3QHApA6jYvewgAA;
domain=.login.microsoftonline.com; path=/; secure;
HttpOnly;fpc=AqAbRMgiP0pMldA4zkrmUY36f-C2AQAAAEzxBdQOAAAA; expires=Wed,
27-Mar-2019 14:23:08 GMT; path=/; secure;
HttpOnly;buid=AQABAAEAAACEfexXxjamQb3OeGQ4Gugva-v_Nlyn-Tpf02RlcpUSwNRWwiDsQefnZ8f2H7B0EiRdWHy-J7zGXejZ1macMiJmALkHcFBkDYfFXzyxWAw6OPIK7OicwCF7wJjiblzZe5MgAA;
expires=Wed, 27-Mar-2019 14:23:08 GMT; path=/; secure; HttpOnly
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - X-DNS-Prefetch-Control=on
2019-02-25 14:23:08,875 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Expires=-1
2019-02-25 14:23:08,876 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Content-Length=27304
2019-02-25 14:23:08,876 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
x-ms-request-id=a48ce162-4579-4ae7-ac93-97e304b05400
2019-02-25 14:23:08,876 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
Link=<https://aadcdn.msauth.net>;
rel=dns-prefetch;<https://aadcdn.msftauth.net>;
rel=dns-prefetch;<https://aadcdn.msftauth.net>; rel=preconnect; crossorigin
2019-02-25 14:23:08,876 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Content-Type=text/html;
charset=utf-8
2019-02-25 14:23:08,876 [main] DEBUG oauth2.AzureADAuthenticator
(AzureADAuthenticator.java:getTokenSingleCall(356)) - AADToken: HTTP connection
to
https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize
failed for getting token from AzureAD. Http response: 200 OK
Content-Type: text/html; charset=utf-8 Content-Length: 27304 Request ID:
a48ce162-4579-4ae7-ac93-97e304b05400 Proxies: none
First 1K of Body:
<!DOCTYPE html>
<html dir="ltr" class="" lang="en">
<head>
<title>Sign in to your account</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0,
maximum-scale=2.0, user-scalable=yes">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<link rel="preconnect" href="https://aadcdn.msftauth.net"; crossorigin>
<meta http-equiv="x-dns-prefetch-control" content="on">
<link rel="dns-prefetch" href="//aadcdn.msftauth.net">
<link rel="dns-prefetch" href="//aadcdn.msauth.net">
<meta name="PageID" content="ConvergedSignIn" />
<meta name="SiteID" content="" />
<meta name="ReqLC" content="1033" />
<meta name="LocLC" content="en-US" />
<noscript>
<meta http-equiv="Refresh" content="0;
URL=https://login.microsoftonline.com/jsdisabled"; />
</noscript>
2019-02-25 14:23:08,879 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(180)) - HttpRequestFailure:
0,,cid=95bbc863-1bcc-483e-b2c4-625bfb2f55fa,rid=,connMs=65,sendMs=0,recvMs=0,sent=0,recv=0,HEAD,https://abfswales1.dfs.core.windows.net/container//?upn=false&action=getAccessControl&timeout=90
org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$UnexpectedResponseException:
HTTP Error 200 ;
url='https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize'OK;
requestId='a48ce162-4579-4ae7-ac93-97e304b05400'; contentType='text/html;
charset=utf-8'; payload'AADToken: HTTP connection to
https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize
failed for getting token from AzureAD. Http response: 200 OK
Content-Type: text/html; charset=utf-8 Content-Length: 27304 Request ID:
a48ce162-4579-4ae7-ac93-97e304b05400 Proxies: none
First 1K of Body:
<!DOCTYPE html>
<html dir="ltr" class="" lang="en">
<head>
<title>Sign in to your account</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0,
maximum-scale=2.0, user-scalable=yes">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<link rel="preconnect" href="https://aadcdn.msftauth.net"; crossorigin>
<meta http-equiv="x-dns-prefetch-control" content="on">
<link rel="dns-prefetch" href="//aadcdn.msftauth.net">
<link rel="dns-prefetch" href="//aadcdn.msauth.net">
<meta name="PageID" content="ConvergedSignIn" />
<meta name="SiteID" content="" />
<meta name="ReqLC" content="1033" />
<meta name="LocLC" content="en-US" />
<noscript>
<meta http-equiv="Refresh" content="0;
URL=https://login.microsoftonline.com/jsdisabled"; />
</noscript>
'
at
org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenSingleCall(AzureADAuthenticator.java:362)
at
org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:267)
at
org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenUsingClientCreds(AzureADAuthenticator.java:96)
at
org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider.refreshToken(ClientCredsTokenProvider.java:58)
at
org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider.getToken(AccessTokenProvider.java:50)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAccessToken(AbfsClient.java:563)
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:151)
at
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:125)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:515)
at
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:498)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getIsNamespaceEnabled(AzureBlobFileSystemStore.java:202)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:467)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:440)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:294)
at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2027)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
ls: HTTP Error 200 ;
url='https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize'OK;
requestId='a48ce162-4579-4ae7-ac93-97e304b05400'; contentType='text/html;
charset=utf-8'; payload'AADToken: HTTP connection to
https://login.microsoftonline.com/b60c9401-2154-40aa-9cff-5e3d1a20085d/oauth2/authorize
failed for getting token from AzureAD. Http response: 200 OK
2019-02-25 14:23:08,887 [shutdown-hook-0] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:close(427)) - AzureBlobFileSystem.close
{code}
> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --------------------------------------------------------------
>
> Key: HADOOP-16068
> URL: https://issues.apache.org/jira/browse/HADOOP-16068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch,
> HADOOP-16068-003.patch, HADOOP-16068-004.patch, HADOOP-16068-005.patch,
> HADOOP-16068-006.patch, HADOOP-16068-007.patch, HADOOP-16068-008.patch,
> HADOOP-16068-009.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind
> the plugins to the specific FS instance. Without that you can't have per FS
> auth
> +add a stub DT plugin for testing, verify that DTs are collected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
