[jira] [Updated] (ARROW-13237) S3 FileSystem doesn't seem to handle redirects

2021-07-15 Thread Alessandro Molina (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Molina updated ARROW-13237:
--
Fix Version/s: (was: 5.0.0)
   6.0.0

> S3 FileSystem doesn't seem to handle redirects
> --
>
> Key: ARROW-13237
> URL: https://issues.apache.org/jira/browse/ARROW-13237
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: C++, Python
>Affects Versions: 4.0.1
>Reporter: Alessandro Molina
>Priority: Major
> Fix For: 6.0.0
>
>
> In some conditions AWS S3 seems to respond with a redirect, but Arrow seems 
> to consider it an error instead of following the redirect.
> For example see
> {code}
> s3, bucket = 
> fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
> print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
> {code}
> The error that you get is
> {code}
>  OSError: When listing objects under key '2011' in bucket 
> 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
> PermanentRedirect Message: The bucket you are attempting to access must be 
> addressed using the specified endpoint. Please send all future requests to 
> this endpoint.
> {code}
> It should probably follow the `PermanentRedirect` instead of choking over it
> IT is also possible to reproduce it using
> {code}
> from pyarrow import fs
> s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
> print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ARROW-13237) S3 FileSystem doesn't seem to handle redirects

2021-07-01 Thread Alessandro Molina (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Molina updated ARROW-13237:
--
Description: 
In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to 
consider it an error instead of following the redirect.

For example see

{code}
s3, bucket = 
fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
{code}

The error that you get is

{code}
 OSError: When listing objects under key '2011' in bucket 
'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
PermanentRedirect Message: The bucket you are attempting to access must be 
addressed using the specified endpoint. Please send all future requests to this 
endpoint.
{code}

It should probably follow the `PermanentRedirect` instead of choking over it

IT is also possible to reproduce it using
{code}
from pyarrow import fs

s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
{code}


  was:
In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to 
consider it an error instead of following the redirect.

For example see

{code}
s3, bucket = 
fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
{code}

The error that you get is

{code}
 OSError: When listing objects under key '2011' in bucket 
'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
PermanentRedirect Message: The bucket you are attempting to access must be 
addressed using the specified endpoint. Please send all future requests to this 
endpoint.
{code}

It should probably follow the `PermanentRedirect` instead of choking over it



> S3 FileSystem doesn't seem to handle redirects
> --
>
> Key: ARROW-13237
> URL: https://issues.apache.org/jira/browse/ARROW-13237
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: C++, Python
>Affects Versions: 4.0.1
>Reporter: Alessandro Molina
>Priority: Major
> Fix For: 5.0.0
>
>
> In some conditions AWS S3 seems to respond with a redirect, but Arrow seems 
> to consider it an error instead of following the redirect.
> For example see
> {code}
> s3, bucket = 
> fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
> print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
> {code}
> The error that you get is
> {code}
>  OSError: When listing objects under key '2011' in bucket 
> 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
> PermanentRedirect Message: The bucket you are attempting to access must be 
> addressed using the specified endpoint. Please send all future requests to 
> this endpoint.
> {code}
> It should probably follow the `PermanentRedirect` instead of choking over it
> IT is also possible to reproduce it using
> {code}
> from pyarrow import fs
> s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
> print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ARROW-13237) S3 FileSystem doesn't seem to handle redirects

2021-07-01 Thread Alessandro Molina (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Molina updated ARROW-13237:
--
Description: 
In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to 
consider it an error instead of following the redirect.

For example see

{code}
s3, bucket = 
fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
{code}

The error that you get is

{code}
 OSError: When listing objects under key '2011' in bucket 
'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
PermanentRedirect Message: The bucket you are attempting to access must be 
addressed using the specified endpoint. Please send all future requests to this 
endpoint.
{code}

It should probably follow the `PermanentRedirect` instead of choking over it


  was:
In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to 
consider it an error instead of following the redirect.

For example see

{code}
s3, _ = fs.S3FileSystem.from_uri("s3://ursa-labs-taxi-data/")
 print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))
{code}

The error that you get is

{code}
 OSError: When listing objects under key '' in bucket '2011': AWS Error [code 
100]: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you 
are attempting to access must be addressed using the specified endpoint. Please 
send all future requests to this endpoint.
{code}

There are two oddities in that error:

1) It should probably follow the `PermanentRedirect`
2) The error states that the bucket name is `2011` while the bucket name was 
`ursa-labs-taxi-data`


> S3 FileSystem doesn't seem to handle redirects
> --
>
> Key: ARROW-13237
> URL: https://issues.apache.org/jira/browse/ARROW-13237
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: C++, Python
>Affects Versions: 4.0.1
>Reporter: Alessandro Molina
>Priority: Major
> Fix For: 5.0.0
>
>
> In some conditions AWS S3 seems to respond with a redirect, but Arrow seems 
> to consider it an error instead of following the redirect.
> For example see
> {code}
> s3, bucket = 
> fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
> print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))
> {code}
> The error that you get is
> {code}
>  OSError: When listing objects under key '2011' in bucket 
> 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: 
> PermanentRedirect Message: The bucket you are attempting to access must be 
> addressed using the specified endpoint. Please send all future requests to 
> this endpoint.
> {code}
> It should probably follow the `PermanentRedirect` instead of choking over it



--
This message was sent by Atlassian Jira
(v8.3.4#803005)