m0g3r opened a new pull request, #3783:
URL: https://github.com/apache/iceberg-python/pull/3783
<!-- Closes #3625 -->
Closes #3625
# Rationale for this change
`FsspecFileIO` remote request signing (`s3.signer=S3V4RestSigner`) silently
stops signing on
current `aiobotocore`/`s3fs`. `_s3()` builds the filesystem first and then
attaches the signer to
`fs.s3`:
```python
fs = S3FileSystem(**s3_fs_kwargs)
for event_name, event_function in register_events.items():
fs.s3.meta.events.unregister(event_name, unique_id=1925)
fs.s3.meta.events.register_last(event_name, event_function,
unique_id=1925)
```
Touching `fs.s3` forces one eager client, but the client that actually
issues requests is created
lazily inside the running event loop by `S3FileSystem.set_session`, and it
does not inherit
handlers registered on that earlier client. Since
`config_kwargs["signature_version"]` is set to
`UNSIGNED`, requests then go out with no `Authorization` header and S3
rejects them with
`InvalidRequest: The authorization mechanism you have provided is not
supported`.
This change registers the signer on an `aiobotocore` session and passes that
session to
`S3FileSystem`, so every client the session creates carries the handler.
`register_last` and the
`unique_id` are kept, so ordering relative to the stock `before-sign.s3`
handlers is unchanged.
The `unregister` call is dropped because a freshly created session has
nothing registered under
that id.
Credit for the diagnosis and the proposed approach goes to the issue
reporter.
## Are these changes tested?
Yes. `test_s3v4_rest_signer_registered_on_session` is added to
`tests/io/test_fsspec.py`. It
builds the `FileIO` with a signer configured, takes the session handed to
`S3FileSystem`, emits
`before-sign.s3` on it, and asserts the REST signer ran — the request URL is
rewritten and an
`Authorization` header is set. On `main` it fails with `KeyError:
'session'`, since no session is
passed today.
The mechanism was also confirmed directly against `s3fs` 2026.4.0 and
`aiobotocore` 3.8.0 by
driving `set_session()` and inspecting the handlers on the client that ends
up issuing requests:
```
register on fs.s3 -> handler present on the real client: False
register on session -> handler present on the real client: True
```
- `make lint` — passes, including mypy
- `make test` — 3804 passed, 3 skipped
Not verified end to end against a live REST catalog with remote signing and
real S3, as that
needs credentials and an S3 endpoint that enforces SigV4; `moto` does not.
The end-to-end
confirmation in the issue report covers that path.
## Are there any user-facing changes?
No API change. Remote signing starts working again on current dependency
versions, so
`FsspecFileIO` users on a modern `aiobotocore` will see requests signed
rather than rejected.
## AI assistance
Claude Code was used to investigate the issue, write the change and the
test, and run the
verification described above. The full diff was reviewed before submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
