subject:"\[jira\] \[Commented\] \(HADOOP\-18826\) abfs getFileStatus\(\/\) fails with \"Value for one of the query parameters specified in the request URI is invalid.\", 400"

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

2023-08-09 Thread Steve Loughran (Jira)



[ 
https://issues.apache.org/jira/browse/HADOOP-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752449#comment-17752449
 ] 

Steve Loughran commented on HADOOP-18826:
-

[~sergey.shabalov]: this bug has been fixed and the next hadoop 3.3.x release 
should include it.

if you need it yourself before then, you are going to have to do your own build 
of a suitable branch with the patch.

> abfs getFileStatus(/) fails with "Value for one of the query parameters 
> specified in the request URI is invalid.", 400
> --
>
> Key: HADOOP-18826
> URL: https://issues.apache.org/jira/browse/HADOOP-18826
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Affects Versions: 3.3.1, 3.3.2, 3.3.5, 3.3.3, 3.3.4, 3.3.6
>Reporter: Sergey Shabalov
>Assignee: Anuj Modi
>Priority: Major
> Fix For: 3.4.0, 3.3.9
>
> Attachments: test_hadoop-azure-3_3_1-FileSystem_getFileStatus - 
> Copy.zip
>
>
> I am using hadoop-azure-3.3.0.jar and have written code:
> {code:java}
> static final String ROOT_DIR = 
> "abfs://ssh-test...@sshadlsgen2.dfs.core.windows.net",
> Configuration config = new Configuration();        
> config.set("fs.defaultFS",ROOT_DIR);        
> config.set("fs.adl.oauth2.access.token.provider.type","ClientCredential");    
>     config.set("fs.adl.oauth2.client.id","");        
> config.set("fs.adl.oauth2.credential","");        
> config.set("fs.adl.oauth2.refresh.url","");        
> config.set("fs.azure.account.key.sshadlsgen2.dfs.core.windows.net",ACCESS_TOKEN);
>         
> config.set("fs.azure.skipUserGroupMetadataDuringInitialization","true");
>   FileSystem fs = FileSystem.get(config);
>   System.out.println( "\nfs:'"+fs.toString()+"'");
>   FileStatus status = fs.getFileStatus(new Path(ROOT_DIR)); // !!! 
> Exception in 3.3.1-*
>   System.out.println( "\nstatus:'"+status.toString()+"'");
>  {code}
> It did work properly till 3.3.1. 
> But in 3.3.1 it fails with exception:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified 
> in the request URI is invalid.", 400, HEAD, 
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>   at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:218)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:181)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:179)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:942)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:924)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:846)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:507)
>  {code}
> I performed some research and found:
> In hadoop-azure-3.3.0.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore{
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
> Line 604: op = 
> client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + 
> AbfsHttpConstants.ROOT_PATH);
>   ...
>   }
>   ...
> } {code}
> and this code produces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs//?upn=false&action=getAccessControl&timeout=90
>   {code}
> There is finalizes slash in path part 
> "...ssh-test-fs{*}{color:#de350b}//{color}{*}?upn=false..." This request does 
> work properly.
> But since hadoop-azure-3.3.1.jar till latest hadoop-azure-3.3.6.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore {
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
>   perfInfo.registerCallee("getAclStatus");
> Line 846:   op = client.getAclStatus(getRelativePath(path));
>   ...
>   }
>   ...
> }
> Line 1492:
> private String getRelativePath(final Path path) {
>   ...
>   return path.toUri().getPath();
> } {code}
> and this code prduces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>  {code}
> There is not finalizes slash in path part "...ssh-test-fs?up

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

2023-07-31 Thread Anuj Modi (Jira)



[ 
https://issues.apache.org/jira/browse/HADOOP-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749146#comment-17749146
 ] 

Anuj Modi commented on HADOOP-18826:


PR for fix: [Hadoop 18826: [ABFS] Fix for Empty Relative Path Issue Leading to 
GetFileStatus("/") failure. by anujmodi2021 · Pull Request #5909 · 
apache/hadoop (github.com)|https://github.com/apache/hadoop/pull/5909]

> abfs getFileStatus(/) fails with "Value for one of the query parameters 
> specified in the request URI is invalid.", 400
> --
>
> Key: HADOOP-18826
> URL: https://issues.apache.org/jira/browse/HADOOP-18826
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Affects Versions: 3.3.1, 3.3.2, 3.3.5, 3.3.3, 3.3.4, 3.3.6
>Reporter: Sergey Shabalov
>Assignee: Anuj Modi
>Priority: Major
> Attachments: test_hadoop-azure-3_3_1-FileSystem_getFileStatus - 
> Copy.zip
>
>
> I am using hadoop-azure-3.3.0.jar and have written code:
> {code:java}
> static final String ROOT_DIR = 
> "abfs://ssh-test...@sshadlsgen2.dfs.core.windows.net",
> Configuration config = new Configuration();        
> config.set("fs.defaultFS",ROOT_DIR);        
> config.set("fs.adl.oauth2.access.token.provider.type","ClientCredential");    
>     config.set("fs.adl.oauth2.client.id","");        
> config.set("fs.adl.oauth2.credential","");        
> config.set("fs.adl.oauth2.refresh.url","");        
> config.set("fs.azure.account.key.sshadlsgen2.dfs.core.windows.net",ACCESS_TOKEN);
>         
> config.set("fs.azure.skipUserGroupMetadataDuringInitialization","true");
>   FileSystem fs = FileSystem.get(config);
>   System.out.println( "\nfs:'"+fs.toString()+"'");
>   FileStatus status = fs.getFileStatus(new Path(ROOT_DIR)); // !!! 
> Exception in 3.3.1-*
>   System.out.println( "\nstatus:'"+status.toString()+"'");
>  {code}
> It did work properly till 3.3.1. 
> But in 3.3.1 it fails with exception:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified 
> in the request URI is invalid.", 400, HEAD, 
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>   at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:218)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:181)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:179)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:942)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:924)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:846)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:507)
>  {code}
> I performed some research and found:
> In hadoop-azure-3.3.0.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore{
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
> Line 604: op = 
> client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + 
> AbfsHttpConstants.ROOT_PATH);
>   ...
>   }
>   ...
> } {code}
> and this code produces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs//?upn=false&action=getAccessControl&timeout=90
>   {code}
> There is finalizes slash in path part 
> "...ssh-test-fs{*}{color:#de350b}//{color}{*}?upn=false..." This request does 
> work properly.
> But since hadoop-azure-3.3.1.jar till latest hadoop-azure-3.3.6.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore {
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
>   perfInfo.registerCallee("getAclStatus");
> Line 846:   op = client.getAclStatus(getRelativePath(path));
>   ...
>   }
>   ...
> }
> Line 1492:
> private String getRelativePath(final Path path) {
>   ...
>   return path.toUri().getPath();
> } {code}
> and this code prduces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>  {code}
> There is not finalizes slash in path part "...ssh-test-fs?upn=false..." It 
> happens because the new code

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

2023-07-26 Thread Steve Loughran (Jira)



[ 
https://issues.apache.org/jira/browse/HADOOP-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747512#comment-17747512
 ] 

Steve Loughran commented on HADOOP-18826:
-

but looking into getRelativePath() is a likely change HADOOP-16916

now: why doesn't anyone else see this?

> abfs getFileStatus(/) fails with "Value for one of the query parameters 
> specified in the request URI is invalid.", 400
> --
>
> Key: HADOOP-18826
> URL: https://issues.apache.org/jira/browse/HADOOP-18826
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Affects Versions: 3.3.1, 3.3.2, 3.3.5, 3.3.3, 3.3.4, 3.3.6
>Reporter: Sergey Shabalov
>Priority: Major
> Attachments: test_hadoop-azure-3_3_1-FileSystem_getFileStatus - 
> Copy.zip
>
>
> I am using hadoop-azure-3.3.0.jar and have written code:
> {code:java}
> static final String ROOT_DIR = 
> "abfs://ssh-test...@sshadlsgen2.dfs.core.windows.net",
> Configuration config = new Configuration();        
> config.set("fs.defaultFS",ROOT_DIR);        
> config.set("fs.adl.oauth2.access.token.provider.type","ClientCredential");    
>     config.set("fs.adl.oauth2.client.id","");        
> config.set("fs.adl.oauth2.credential","");        
> config.set("fs.adl.oauth2.refresh.url","");        
> config.set("fs.azure.account.key.sshadlsgen2.dfs.core.windows.net",ACCESS_TOKEN);
>         
> config.set("fs.azure.skipUserGroupMetadataDuringInitialization","true");
>   FileSystem fs = FileSystem.get(config);
>   System.out.println( "\nfs:'"+fs.toString()+"'");
>   FileStatus status = fs.getFileStatus(new Path(ROOT_DIR)); // !!! 
> Exception in 3.3.1-*
>   System.out.println( "\nstatus:'"+status.toString()+"'");
>  {code}
> It did work properly till 3.3.1. 
> But in 3.3.1 it fails with exception:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified 
> in the request URI is invalid.", 400, HEAD, 
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>   at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:218)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:181)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:179)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:942)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:924)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:846)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:507)
>  {code}
> I performed some research and found:
> In hadoop-azure-3.3.0.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore{
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
> Line 604: op = 
> client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + 
> AbfsHttpConstants.ROOT_PATH);
>   ...
>   }
>   ...
> } {code}
> and this code produces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs//?upn=false&action=getAccessControl&timeout=90
>   {code}
> There is finalizes slash in path part 
> "...ssh-test-fs{*}{color:#de350b}//{color}{*}?upn=false..." This request does 
> work properly.
> But since hadoop-azure-3.3.1.jar till latest hadoop-azure-3.3.6.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore {
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
>   perfInfo.registerCallee("getAclStatus");
> Line 846:   op = client.getAclStatus(getRelativePath(path));
>   ...
>   }
>   ...
> }
> Line 1492:
> private String getRelativePath(final Path path) {
>   ...
>   return path.toUri().getPath();
> } {code}
> and this code prduces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>  {code}
> There is not finalizes slash in path part "...ssh-test-fs?upn=false..." It 
> happens because the new code "path.toUri().getPath();" produces empty string.
> This request fails with message:
> {code:java}
> Caused by: Operation failed: "Valu

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

2023-07-26 Thread Steve Loughran (Jira)



[ 
https://issues.apache.org/jira/browse/HADOOP-18826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747509#comment-17747509
 ] 

Steve Loughran commented on HADOOP-18826:
-

based on your analysis, HADOOP-16612 is the cause

> abfs getFileStatus(/) fails with "Value for one of the query parameters 
> specified in the request URI is invalid.", 400
> --
>
> Key: HADOOP-18826
> URL: https://issues.apache.org/jira/browse/HADOOP-18826
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/azure
>Affects Versions: 3.3.1, 3.3.2, 3.3.5, 3.3.3, 3.3.4, 3.3.6
>Reporter: Sergey Shabalov
>Priority: Major
> Attachments: test_hadoop-azure-3_3_1-FileSystem_getFileStatus - 
> Copy.zip
>
>
> I am using hadoop-azure-3.3.0.jar and have written code:
> {code:java}
> static final String ROOT_DIR = 
> "abfs://ssh-test...@sshadlsgen2.dfs.core.windows.net",
> Configuration config = new Configuration();        
> config.set("fs.defaultFS",ROOT_DIR);        
> config.set("fs.adl.oauth2.access.token.provider.type","ClientCredential");    
>     config.set("fs.adl.oauth2.client.id","");        
> config.set("fs.adl.oauth2.credential","");        
> config.set("fs.adl.oauth2.refresh.url","");        
> config.set("fs.azure.account.key.sshadlsgen2.dfs.core.windows.net",ACCESS_TOKEN);
>         
> config.set("fs.azure.skipUserGroupMetadataDuringInitialization","true");
>   FileSystem fs = FileSystem.get(config);
>   System.out.println( "\nfs:'"+fs.toString()+"'");
>   FileStatus status = fs.getFileStatus(new Path(ROOT_DIR)); // !!! 
> Exception in 3.3.1-*
>   System.out.println( "\nstatus:'"+status.toString()+"'");
>  {code}
> It did work properly till 3.3.1. 
> But in 3.3.1 it fails with exception:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified 
> in the request URI is invalid.", 400, HEAD, 
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>   at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:218)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:181)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
>  at 
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:179)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:942)
>  at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAclStatus(AbfsClient.java:924)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.getFileStatus(AzureBlobFileSystemStore.java:846)
>  at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.getFileStatus(AzureBlobFileSystem.java:507)
>  {code}
> I performed some research and found:
> In hadoop-azure-3.3.0.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore{
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
> Line 604: op = 
> client.getAclStatus(AbfsHttpConstants.FORWARD_SLASH + 
> AbfsHttpConstants.ROOT_PATH);
>   ...
>   }
>   ...
> } {code}
> and this code produces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs//?upn=false&action=getAccessControl&timeout=90
>   {code}
> There is finalizes slash in path part 
> "...ssh-test-fs{*}{color:#de350b}//{color}{*}?upn=false..." This request does 
> work properly.
> But since hadoop-azure-3.3.1.jar till latest hadoop-azure-3.3.6.jar we see:
> {code:java}
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore {
>   ...
>   public FileStatus getFileStatus(final Path path) throws IOException {
>   ...
>   perfInfo.registerCallee("getAclStatus");
> Line 846:   op = client.getAclStatus(getRelativePath(path));
>   ...
>   }
>   ...
> }
> Line 1492:
> private String getRelativePath(final Path path) {
>   ...
>   return path.toUri().getPath();
> } {code}
> and this code prduces REST request:
> {code:java}
> https://sshadlsgen2.dfs.core.windows.net/ssh-test-fs?upn=false&action=getAccessControl&timeout=90
>  {code}
> There is not finalizes slash in path part "...ssh-test-fs?upn=false..." It 
> happens because the new code "path.toUri().getPath();" produces empty string.
> This request fails with message:
> {code:java}
> Caused by: Operation failed: "Value for one of the query parameters specified 
> in the req

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

[jira] [Commented] (HADOOP-18826) abfs getFileStatus(/) fails with "Value for one of the query parameters specified in the request URI is invalid.", 400

4 matches

Site Navigation

Mail list logo

Footer information