[
https://issues.apache.org/jira/browse/ARROW-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joris Van den Bossche updated ARROW-5072:
-----------------------------------------
Labels: parquet (was: )
> [Python] write_table fails silently on S3 errors
> ------------------------------------------------
>
> Key: ARROW-5072
> URL: https://issues.apache.org/jira/browse/ARROW-5072
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.12.1
> Environment: Python 3.6.8
> Reporter: Paul George
> Priority: Minor
> Labels: parquet
>
> {{pyarrow==0.12.1}}
> *pyarrow.parquet.write_table* called with where=S3File(...) fails silently
> when encountering errors while writing to S3 (in the example below, boto3 is
> raising a NoSuchBucket exception). However, instead of using S3File(),
> calling write_table with where=_<filepath>_ and with
> filesystem=S3FileSystem() does *not* fail silently and raises, as is expected.
> h4. Code/Repro
>
> {code:java}
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> from s3fs import S3File, S3FileSystem
> df = pd.DataFrame({'col0': []})
> s3_filepath = 's3://some-bogus-bucket/df.parquet'
> print('>> test 1')
> try:
> # use S3File --> fails silently
> pq.write_table(pa.Table.from_pandas(df.copy()),
> S3File(S3FileSystem(), s3_filepath, mode='wb'))
> except Exception:
> print('>>>> Exception raised!')
> else:
> print('>>>> Exception **NOT** raised!')
> print('>> test 2')
> try:
> # use filepath and S3FileSystem --> raises Exception, as expected
> pq.write_table(pa.Table.from_pandas(df.copy()),
> s3_filepath,
> filesystem=S3FileSystem())
> except Exception:
> print('>>>> Exception raised!')
> else:
> print('>>>> Exception **NOT** raised!'){code}
>
> h4.
> h4. Output
> {code:java}
> >> test 1
> Exception ignored in: <bound method S3File.__del__ of <S3File
> some-bogus-bucket/df.parquet>>
> Traceback (most recent call last):
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in
> __del__
> self.close()
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in
> close
> raise_from(IOError('Write failed: %s' % self.path), e)
> File "<string>", line 3, in raise_from
> OSError: Write failed: some-bogus-bucket/df.parquet
> >>>> Exception **NOT** raised!
> >> test 2
> >>>> Exception raised!
> Exception ignored in: <bound method S3File.__del__ of <S3File
> some-bogus-bucket/df.parquet>>
> Traceback (most recent call last):
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in
> __del__
> self.close()
> File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in
> close
> raise_from(IOError('Write failed: %s' % self.path), e)
> File "<string>", line 3, in raise_from
> OSError: Write failed: some-bogus-bucket/df.parquet
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
