mdimas commented on issue #50188:
URL: https://github.com/apache/arrow/issues/50188#issuecomment-6380908582
Reproduced on **pyarrow 26.0.0** (manylinux x86_64 wheel), so this still
affects the latest release. Our environment isn't quite the plain-Docker setup
you asked for, but it's close: a Linux pod on EKS using IRSA, with the
`python:3.12-slim` (Debian) base image and Python 3.12.15. The pod gets the
standard web-identity environment from the EKS webhook: `AWS_ROLE_ARN`,
`AWS_WEB_IDENTITY_TOKEN_FILE`, `AWS_REGION`, `AWS_DEFAULT_REGION` and
`AWS_STS_REGIONAL_ENDPOINTS=regional`.
**Script:**
```python
import sys, time
import pyarrow.fs as fs
kwargs = {}
if "--explicit" in sys.argv:
# Same role's credentials, resolved by boto3 and passed in explicitly
import boto3
c = boto3.Session().get_credentials().get_frozen_credentials()
kwargs = dict(access_key=c.access_key, secret_key=c.secret_key,
session_token=c.token)
s3 = fs.S3FileSystem(region="us-west-2", **kwargs)
infos = s3.get_file_info(fs.FileSelector("<bucket>", recursive=False))
print(f"listed {len(infos)} entries", flush=True)
if "--finalize" in sys.argv:
fs.finalize_s3()
print("finalize_s3 returned", flush=True)
print("main done", flush=True)
```
Each case ran under `timeout 60 python repro.py [flag]`, in the same pod:
| Credentials | Script output | Process exit |
|---|---|---|
| Web identity (IRSA env as above), run twice | `main done` after ~0.1s |
**hangs**, killed by `timeout` (exit 124) both times |
| Web identity + `--finalize` | `listed ...`, but `finalize_s3 returned`
never printed | **hangs inside `finalize_s3()`** (exit 124) |
| `env -u AWS_ROLE_ARN -u AWS_WEB_IDENTITY_TOKEN_FILE` (falls back to the
EC2 instance profile) | `main done` | exits in ~1s (exit 0) |
| `--explicit` (the same role's temporary credentials, passed to
`S3FileSystem`) | `main done` | exits in ~1s (exit 0) |
So a single request using `STS_WEB_IDENTITY` credentials is enough to
trigger it. There's no long-running process or credential refresh involved, and
it doesn't happen with instance-profile credentials or with explicit
credentials for the same role.
**AWS SDK trace** (`fs.initialize_s3(fs.S3LogLevel.Trace)`, web-identity
case). Both requests succeed, and there are no errors apart from the usual
missing `~/.aws/config` and `~/.aws/credentials`:
```
[DEBUG] AuthCredentialsProvider (id=0x...) STS_WEB_IDENTITY credentials
provider query received http status code 200
[DEBUG] CurlHttpClient Returned http response code 200
```
After `main done`, the trace shows the address cache for
`sts.us-west-2.amazonaws.com` refreshing every second. About 30s after the
request, it shuts down ("purging address ... from the cache due to cache
eviction or shutdown"). After that the log is silent until the process is
killed.
**Native stack of the hung main thread** (`py-spy dump --native`, from a
production job hitting the same hang; the other threads are idle):
```
Thread (idle): "MainThread"
syscall (libc.so.6)
std::__atomic_futex_unsigned_base::_M_futex_wait_until
(libstdc++.so.6.0.33)
Aws::Crt::Io::ClientBootstrap::~ClientBootstrap
(pyarrow/libarrow_s3.so.2600)
Aws::SetDefaultClientBootstrap (pyarrow/libarrow_s3.so.2600)
Aws::CleanupCrt (pyarrow/libarrow_s3.so.2600)
Aws::ShutdownAPI (pyarrow/libarrow_s3.so.2600)
arrow::fs::EnsureS3Finalized (pyarrow/libarrow_s3.so.2600)
ensure_s3_finalized (pyarrow/_s3fs.cpython-312-x86_64-linux-gnu.so)
```
This matches the original report. Together with the explicit-credentials
control, it suggests the web-identity credentials provider's own STS client is
still holding the default `ClientBootstrap` when `ShutdownAPI` runs. That would
also fit the aws-sdk-cpp maintainer's explanation in aws/aws-sdk-cpp#2769.
**Workaround we're using for now:** our jobs end with `os._exit()` after
flushing logs, which skips the atexit handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
