[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902117#comment-16902117
]
Steve Loughran commented on HADOOP-16138:
-
Gabor,
you active here?
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905158#comment-16905158
]
Gabor Bota commented on HADOOP-16138:
-
Not recently, but I plan to fix this this week.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908062#comment-16908062
]
Gabor Bota commented on HADOOP-16138:
-
So I've created a test for it (new test class for testing CLI with ABFS).
The test was not failing. The output is:
{{mkdir:
`abfs://nonexistent-3ab66e98-66dc-4e7b-879d-16e323bd2...@mycontainer.dfs.core.windows.net/':
File exists}}
Next, I tried to run from a {{dist}}, where the output was the same. So I guess
this got fixed, but we could add the test I've created for this - so it's ready
for review!
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908308#comment-16908308
]
Gabor Bota commented on HADOOP-16138:
-
Based on offline discussion with [~mackrorysd] we agreed that this output is
not what we would like to see.
Something like "{{The container does not exist: [nameofthecontainer].}}" is way
better than what we currently have.
I'll update my PR accordingly.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913415#comment-16913415
]
Gabor Bota commented on HADOOP-16138:
-
I found out what's happening. We auto-create all containers during the test
run, so I should turn that feature off for this test. I forgot to do that, and
so I ended up a lot of random containers in the testing account.
In
{{org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest#AbstractAbfsIntegrationTest}}
we do a
{code:java}
abfsConfig.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION,
true);
{code}
which is not good, so I'll create a patch where this can be passed as a
parameter.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797191#comment-16797191
]
Gabor Bota commented on HADOOP-16138:
-
Some update on this: I managed to reproduce the issue via CLI but not in a unit
test. It seems like the call sequence is quite different if you call {{mkdirs}}
on {{AzureBlobFileSystem}} directly and via CLI, and something goes wrong when
it is called as a CLI command.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797327#comment-16797327
]
Gabor Bota commented on HADOOP-16138:
-
So the issue is that in fs.shell.Command.java:250 we call
{{PathData.expandAsGlob}} on the path string, which returns null (the
getFileStatus call will return null) and then there's a call stack:
{noformat}
getFileStatus:434, AzureBlobFileSystem (org.apache.hadoop.fs.azurebfs)
exists:1695, FileSystem (org.apache.hadoop.fs)
processNonexistentPath:76, Mkdir (org.apache.hadoop.fs.shell)
processArgument:288, Command (org.apache.hadoop.fs.shell)
processArguments:270, Command (org.apache.hadoop.fs.shell)
processRawArguments:120, FsCommand (org.apache.hadoop.fs.shell)
run:177, Command (org.apache.hadoop.fs.shell)
run:327, FsShell (org.apache.hadoop.fs)
run:76, ToolRunner (org.apache.hadoop.util)
run:90, ToolRunner (org.apache.hadoop.util)
main:390, FsShell (org.apache.hadoop.fs)
{noformat}
which will end up calling a
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem#makeQualified with null
parameter which will end up in a stack:
{noformat}
checkPath:712, FileSystem (org.apache.hadoop.fs)
makeQualified:602, FileSystem (org.apache.hadoop.fs)
makeQualified:466, AzureBlobFileSystem (org.apache.hadoop.fs.azurebfs)
getFileStatus:436, AzureBlobFileSystem (org.apache.hadoop.fs.azurebfs)
exists:1695, FileSystem (org.apache.hadoop.fs)
processNonexistentPath:76, Mkdir (org.apache.hadoop.fs.shell)
processArgument:288, Command (org.apache.hadoop.fs.shell)
processArguments:270, Command (org.apache.hadoop.fs.shell)
processRawArguments:120, FsCommand (org.apache.hadoop.fs.shell)
run:177, Command (org.apache.hadoop.fs.shell)
run:327, FsShell (org.apache.hadoop.fs)
run:76, ToolRunner (org.apache.hadoop.util)
run:90, ToolRunner (org.apache.hadoop.util)
main:390, FsShell (org.apache.hadoop.fs)
{noformat}
so it will call {{URI uri = path.toUri();}} inside checkPath, but path will be
null.
The solution is to check in
{{org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem#getFileStatus}} if Path f
is null, instead of directly calling makeQualified.
I'll create a pr soon with this.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935762#comment-16935762
]
Gabor Bota commented on HADOOP-16138:
-
+1 by [~ste...@apache.org] on PR #1302. Committing.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935763#comment-16935763
]
Hudson commented on HADOOP-16138:
-
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17356 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17356/])
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE
(github: rev aa664d72595ddfcb1a1bf082381bb222e59db354)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Mkdir.java
* (add)
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCLI.java
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937377#comment-16937377
]
Ayush Saxena commented on HADOOP-16138:
---
Hi [~ste...@apache.org] [~gabor.bota]
I guess this change breaks
{{TestHDFSCLI}} and {{TestDFSShell}}
Since it changed the text in the exception :
{code:java}
-throw new PathNotFoundException(itemParentPath.toString());
+throw new PathNotFoundException(String.format(
+"mkdir failed for path: %s. Item parent path not found: %s.",
+itemPath.toString(), itemParentPath.toString()));
}
{code}
For reference :
https://builds.apache.org/job/PreCommit-HDFS-Build/27958/testReport/
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937551#comment-16937551
]
Gabor Bota commented on HADOOP-16138:
-
That is true. Sorry [~ayushtkn].
The way I plan to fix it: create an HDFS jira where I `revert` this change in
the sense that I will create a PR with the original log. No need for the
additional logging what we added.
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775313#comment-16775313
]
Steve Loughran commented on HADOOP-16138:
-
{code}
2019-02-22 16:00:05,804 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:initialize(101)) - Initializing AzureBlobFileSystem
for abfs://contain...@abfswales1.dfs.core.windows.net/
2019-02-22 16:00:06,082 [main] DEBUG services.AbfsClientThrottlingIntercept
(AbfsClientThrottlingIntercept.java:initializeSingleton(62)) - Client-side
throttling is enabled for the ABFS file system.
2019-02-22 16:00:06,102 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:getFileStatus(434)) -
AzureBlobFileSystem.getFileStatus path:
abfs://contain...@abfswales1.dfs.core.windows.net/
2019-02-22 16:00:06,102 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:performAbfsAuthCheck(1101)) - ABFS authorizer is not
initialized. No authorization check will be performed.
2019-02-22 16:00:06,102 [main] DEBUG azurebfs.AzureBlobFileSystemStore
(AzureBlobFileSystemStore.java:getIsNamespaceEnabled(200)) - Get root ACL status
2019-02-22 16:00:06,132 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(152)) - Signing request with
shared key
2019-02-22 16:00:06,136 [main] DEBUG services.SharedKeyCredentials
(SharedKeyCredentials.java:signRequest(96)) - Signing request with timestamp of
Fri, 22 Feb 2019 16:00:06 GMT and signature SharedKey /* snip */
2019-02-22 16:00:06,136 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Response Headers
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Accept-Charset=utf-8
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - x-ms-version=2018-11-09
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Accept=application/json,
application/octet-stream
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - x-ms-date=Fri, 22 Feb 2019
16:00:06 GMT
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - User-Agent=Azure Blob
FS/3.3.0-SNAPSHOT (JavaJRE 1.8.0_121; MacOSX 10.14; openssl-1.0) unknown
2019-02-22 16:00:06,137 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
x-ms-client-request-id=d4484925-a273-482e-8aa9-e0fdef6cdf82
2019-02-22 16:00:06,138 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Content-Type=
2019-02-22 16:00:06,277 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(51)) - Request Headers
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - HTTP Response=HTTP/1.1 404 The
specified filesystem does not exist.
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - x-ms-version=2018-11-09
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Server=Windows-Azure-HDFS/1.0
Microsoft-HTTPAPI/2.0
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
x-ms-error-code=FilesystemNotFound
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) -
x-ms-request-id=d4459a80-901f-0015-41c7-ca989000
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsIoUtils
(AbfsIoUtils.java:dumpHeadersToDebugLog(54)) - Date=Fri, 22 Feb 2019 16:00:05
GMT
2019-02-22 16:00:06,279 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(202)) - HttpRequest:
404,,cid=d4484925-a273-482e-8aa9-e0fdef6cdf82,rid=d4459a80-901f-0015-41c7-ca989000,connMs=26,sendMs=0,recvMs=138,sent=0,recv=0,HEAD,https://abfswales1.dfs.core.windows.net/container2//?upn=false&action=getAccessControl&timeout=90
2019-02-22 16:00:06,285 [main] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:getFileStatus(434)) -
AzureBlobFileSystem.getFileStatus path: null
2019-02-22 16:00:06,286 [main] DEBUG fs.FsShell (FsShell.java:run(344)) - Error
java.lang.NullPointerException
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:712)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:602)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.makeQualified(AzureBlobFileSystem.java:466)
at
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:436)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1695)
at
org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:76)
at org.apache.hadoo
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775325#comment-16775325
]
Steve Loughran commented on HADOOP-16138:
-
If you try to create a subdir then it fails more meaningfully
{code}
2019-02-22 16:05:43,642 [main] DEBUG services.AbfsClient
(AbfsRestOperation.java:executeHttpOperation(202)) - HttpRequest:
404,,cid=ba8b4d49-e05f-493e-97f4-bd1bde48f845,rid=f6ca5c36-f01f-002c-30c8-cad83400,connMs=0,sendMs=0,recvMs=23,sent=0,recv=0,HEAD,https://abfswales1.dfs.core.windows.net/container2//?upn=false&action=getAccessControl&timeout=90
mkdir: `abfs://contain...@abfswales1.dfs.core.windows.net/': No such file or
directory
2019-02-22 16:05:43,644 [shutdown-hook-0] DEBUG azurebfs.AzureBlobFileSystem
(AzureBlobFileSystem.java:close(427)) - AzureBlobFileSystem.close
{code}
One thing to consider here is FNFE the right thing to raise when looking for
the root directory -as its possible that this is being caught during mkdir as
"we should create something". Maybe if the error handling can pick up the
`x-ms-error-code=FilesystemNotFound` error, it could generate a different
response
FWIW, a call of "hadoop fs -mkdir s3a://hwdev-steve-new2/" does fail
mkdir: Bucket hwdev-steve-new2 does not exist,
-the existence check in S3AFS is just throwing an FNFE, for mkdir to handle.
Also, "hadoop fs mkdir" is clearly broken when it is told to create the root
directory.
Maybe both the stores would benefit from a new "ObjectStoreDoesNotExist"
exception (Extends FNFE) for a clearer message; we could even think about
having FS shell handle it specially too
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE
[
https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978918#comment-16978918
]
Hudson commented on HADOOP-16138:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17680 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17680/])
HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace
(iwasakims: rev de38045021875eac0558f06117942f5b849b4975)
* (edit)
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCLI.java
> hadoop fs mkdir / of nonexistent abfs container raises NPE
> --
>
> Key: HADOOP-16138
> URL: https://issues.apache.org/jira/browse/HADOOP-16138
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 3.3.0
>
>
> If you try to do a mkdir on the root of a nonexistent container, you get an
> NPE
> {code}
> hadoop fs -mkdir abfs://contain...@abfswales1.dfs.core.windows.net/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
