I don't have great numbers on when it is failing and when not but I believe I have only seen this for objects larger than 100MB.
Our local S3 compatible object store (via ceph) is in the same datacenter where I am running arrow code and the least network connection involved should be 25Gbps. The rgw gateways (s3 endpoints) can get pretty loaded and I could see a second or 2 wait happening from time to time in queue waiting to get serviced. Hopefully the api is not doing a list on the bucket in this operation, that can be really slow. I think being able to configure timeouts and retries would be a great addition. thanks again, Luke On Mon, Mar 29, 2021 at 11:00 AM Antoine Pitrou <[email protected]> wrote: > On Mon, 29 Mar 2021 10:52:04 -0400 > Luke <[email protected]> wrote: > > No firewall in this setup. And no errors when the keys are small that I > > have seen. Also, using boto3 or requests there are no issues that I can > > tell (although it may be suppressing timeouts and retries). > > How large is the file that gives an error, and how fast is your > connection to the S3 server? > Also, after how long do you get the timeout error? > > Regards > > Antoine. > > > > > > So you are saying the 'AWS Error [code 99]: curlCode: 28, Timeout was > > reached' is coming from the server? Interesting, we haven't seen that > > before but our access before this was with requests or s3fs which uses > > boto3 under the hood and must handle this if so. > > > > thanks, will explore more, > > Luke > > > > On Mon, Mar 29, 2021 at 9:54 AM Antoine Pitrou <[email protected]> > wrote: > > > > > > > > Hi Luke, > > > > > > Given the error message, my intuition is that the timeout is on the > > > server side. Arrow does not try to set any timeouts on S3 connections. > > > > > > Note that this message ("When reading information") happens *before* > > > reading the file data, simply when trying to read the file length. So > > > perhaps something is weird in your network configuration (is a firewall > > > blocking packets?). > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > > On Sat, 27 Mar 2021 10:44:53 -0400 > > > Luke <[email protected]> wrote: > > > > I have a local S3 compatible object store (using ceph) and am trying > to > > > use > > > > the pyarrow fs interface. This seems to work well except on larger > > > objects > > > > I am getting unhandled exceptions. Is there a way to currently tune > the > > > > timeouts or retries? Here is the kind of code and error I am seeing: > > > > > > > > from pyarrow import fs > > > > > > > > > > > > > > > > s3 = > > > > > > > > fs.S3FileSystem(access_key=my_ak,secret_key=my_sk,endpoint_override=my_endpoint,scheme='http') > > > > > > > > > raw = s3.open_input_stream('test_bucket/example_key').readall() > > > > > > > > > > > > > > > > File "pyarrow/_fs.pyx", line 621, in > > > > pyarrow._fs.FileSystem.open_input_stream > > > > > > > > File "pyarrow/error.pxi", line 122, in > > > > pyarrow.lib.pyarrow_internal_check_status > > > > > > > > File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status > > > > > > > > OSError: When reading information for key 'example_key' in bucket > > > > 'test_bucket': AWS Error [code 99]: curlCode: 28, Timeout was reached > > > > > > > > > > > > > > > > -- > > > > > > > > install details: > > > > > > > > python: python 3.8.6 > > > > > > > > OS: linux, redhat 7.7 > > > > > > > > pyarrow version: 3.0.0 > > > > > > > > > > > > thanks for the help, > > > > > > > > Luke > > > > > > > > > > > > > > > > > > > > > >
