[ https://issues.apache.org/jira/browse/ARROW-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche updated ARROW-5072: ----------------------------------------- Labels: parquet (was: ) > [Python] write_table fails silently on S3 errors > ------------------------------------------------ > > Key: ARROW-5072 > URL: https://issues.apache.org/jira/browse/ARROW-5072 > Project: Apache Arrow > Issue Type: Bug > Components: Python > Affects Versions: 0.12.1 > Environment: Python 3.6.8 > Reporter: Paul George > Priority: Minor > Labels: parquet > > {{pyarrow==0.12.1}} > *pyarrow.parquet.write_table* called with where=S3File(...) fails silently > when encountering errors while writing to S3 (in the example below, boto3 is > raising a NoSuchBucket exception). However, instead of using S3File(), > calling write_table with where=_<filepath>_ and with > filesystem=S3FileSystem() does *not* fail silently and raises, as is expected. > h4. Code/Repro > > {code:java} > import pandas as pd > import pyarrow as pa > import pyarrow.parquet as pq > from s3fs import S3File, S3FileSystem > df = pd.DataFrame({'col0': []}) > s3_filepath = 's3://some-bogus-bucket/df.parquet' > print('>> test 1') > try: > # use S3File --> fails silently > pq.write_table(pa.Table.from_pandas(df.copy()), > S3File(S3FileSystem(), s3_filepath, mode='wb')) > except Exception: > print('>>>> Exception raised!') > else: > print('>>>> Exception **NOT** raised!') > print('>> test 2') > try: > # use filepath and S3FileSystem --> raises Exception, as expected > pq.write_table(pa.Table.from_pandas(df.copy()), > s3_filepath, > filesystem=S3FileSystem()) > except Exception: > print('>>>> Exception raised!') > else: > print('>>>> Exception **NOT** raised!'){code} > > h4. > h4. Output > {code:java} > >> test 1 > Exception ignored in: <bound method S3File.__del__ of <S3File > some-bogus-bucket/df.parquet>> > Traceback (most recent call last): > File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in > __del__ > self.close() > File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in > close > raise_from(IOError('Write failed: %s' % self.path), e) > File "<string>", line 3, in raise_from > OSError: Write failed: some-bogus-bucket/df.parquet > >>>> Exception **NOT** raised! > >> test 2 > >>>> Exception raised! > Exception ignored in: <bound method S3File.__del__ of <S3File > some-bogus-bucket/df.parquet>> > Traceback (most recent call last): > File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in > __del__ > self.close() > File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in > close > raise_from(IOError('Write failed: %s' % self.path), e) > File "<string>", line 3, in raise_from > OSError: Write failed: some-bogus-bucket/df.parquet > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)