ovcharenko opened a new issue, #39703:
URL: https://github.com/apache/arrow/issues/39703
### Describe the bug, including details regarding any error messages,
version, and platform.
When running such a simple test in Docker I'm getting `Segmentation fault`
with PyArrow v14.0.2. The same test works fine in a versions bellow.
**Sample test**:
```python
import concurrent.futures
import pyarrow.parquet as pq
from time import time
feeds = [
"file1",
"file2",
...
"fileN",
]
def try_feed_load(feed: str):
start = time()
pq.read_table(f"s3://bucket_name/{feed}", filters=None)
end = time()
print(f"{feed} took {end - start} seconds")
with concurrent.futures.ThreadPoolExecutor(max_workers=6) as pool_executor:
for collected_data in pool_executor.map(try_feed_load, feeds):
pass
```
**To repeat**:
1. Start a container:
`docker run --rm -it --init --ulimit core=-1 --mount
type=bind,source=/tmp/,target=/tmp/ -v /root/work:/work --entrypoint /bin/bash
amd64/python:3.11-slim`
2. Install most recent PyArrow package:
```shell
cd /work
python3 -m venv .venv_recent
. .venv_recent/bin/activate
pip install pyarrow # That will install
pyarrow-14.0.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata
```
3. Run a test. It fails immediately:
```shell
python test2.py
Segmentation fault (core dumped)
```
**Workarounds**:
1. Change the script to run at most 1 worker
2. Downgrade to PyArrow v14.0.1
**Failure details**:
```gdb
Thread 67 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f8bf97956c0 (LWP 943)]
0x00007f8ca273cd50 in Aws::Utils::Threading::ReaderWriterLock::LockReader()
() from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
(gdb) bt
#0 0x00007f8ca273cd50 in
Aws::Utils::Threading::ReaderWriterLock::LockReader() () from
/usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#1 0x00007f8ca275ee62 in
Aws::Config::ConfigAndCredentialsCacheManager::GetConfigProfile(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) const
() from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#2 0x00007f8ca275f103 in
Aws::Config::GetCachedConfigProfile(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#3 0x00007f8ca2761736 in
Aws::Auth::STSAssumeRoleWebIdentityCredentialsProvider::STSAssumeRoleWebIdentityCredentialsProvider()
()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#4 0x00007f8ca2763866 in
Aws::Auth::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain()
() from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#5 0x00007f8ca1fbf579 in
arrow::fs::S3Options::ConfigureDefaultCredentials() () from
/usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#6 0x00007f8ca1fceac0 in arrow::fs::S3Options::FromUri(arrow::internal::Uri
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#7 0x00007f8ca12fa070 in arrow::fs::(anonymous
namespace)::FileSystemFromUriReal(arrow::internal::Uri const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&, arrow::io::IOContext const&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >*) ()
from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#8 0x00007f8ca12faa80 in
arrow::fs::FileSystemFromUri(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, arrow::io::IOContext
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*) () from
/usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#9 0x00007f8ca12fab7c in
arrow::fs::FileSystemFromUriOrPath(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&, arrow::io::IOContext
const&, std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >*) () from
/usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#10 0x00007f8ca12facf3 in
arrow::fs::FileSystemFromUriOrPath(std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
>*) () from /usr/local/lib/python3.11/site-packages/pyarrow/libarrow.so.1400
#11 0x00007f8bfa118712 in
__pyx_pf_7pyarrow_3_fs_10FileSystem_2from_uri(_object*) () from
/usr/local/lib/python3.11/site-packages/pyarrow/_fs.cpython-311-x86_64-linux-gnu.so
#12 0x00007f8ca50d5843 in PyObject_Vectorcall () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#13 0x00007f8ca50ca92b in _PyEval_EvalFrameDefault () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#14 0x00007f8ca50c94ba in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#15 0x00007f8ca50c4a5d in _PyObject_FastCallDictTstate () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#16 0x00007f8ca50e6244 in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#17 0x00007f8ca50c3721 in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#18 0x00007f8ca50c35cf in _PyObject_MakeTpCall () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#19 0x00007f8ca50ca92b in _PyEval_EvalFrameDefault () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#20 0x00007f8ca50c94ba in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#21 0x00007f8ca50ccab1 in _PyEval_EvalFrameDefault () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#22 0x00007f8ca50c94ba in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#23 0x00007f8ca50ccab1 in _PyEval_EvalFrameDefault () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#24 0x00007f8ca50c94ba in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#25 0x00007f8ca50f2e6d in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#26 0x00007f8ca516c3c4 in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#27 0x00007f8ca516c354 in ?? () from
/usr/local/bin/../lib/libpython3.11.so.1.0
#28 0x00007f8ca4d72044 in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:442
#29 0x00007f8ca4df1880 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:100
(gdb)
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
