o-nikolas commented on code in PR #75997:
URL: https://github.com/apache/airflow/pull/75997#discussion_r4426294030
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/duckdb.py:
##########
@@ -196,6 +200,25 @@ def get_s3_endpoint_url(self) -> str | None:
return explicit
return self.connection_extra.get("endpoint_url") or
self.aws_hook.conn_config.endpoint_url
+ def get_conn(self) -> DuckDBPyConnection:
+ """Return a DuckDB connection, and log the AWS identity once to help
debug S3 ``HTTP 403`` errors."""
+ conn = super().get_conn()
+ if self._aws_identity_looked_up:
+ return conn
+ self._aws_identity_looked_up = True
+ try:
+ if self.credential_strategy != "none" and not
self.get_s3_endpoint_url():
+ # Keeps an unreachable STS endpoint from delaying the task by
more than a few seconds.
+ bounded = Config(connect_timeout=2, read_timeout=2,
retries={"total_max_attempts": 1})
+ config = self.aws_hook.config.merge(bounded)
+ sts_hook = StsHook(
+ aws_conn_id=self.aws_conn_id,
region_name=self.get_region_name(), config=config
Review Comment:
Good catch. For `credential_chain` the lookup now uses `aws_conn_id=None`,
so boto3 follows its default chain like DuckDB does. `config` still uses the
connection since those are the exact creds that go into the secret. I tested it
against a fake local STS with different keys on the connection vs the
environment, and each strategy signs with the keys DuckDB would use. The only
gap left is a custom `credential_chain` order, where DuckDB could still land on
something different from boto3's default chain, but that feels rare enough to
leave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
