[
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bjorn Olsen updated AIRFLOW-6072:
---------------------------------
Description:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
To set an outbound proxy is then as simple as adding this to your extra_config:
{code:java}
{ .. ,
"config":{ "proxies": {
"http": "http://myproxy:8080";,
"https": "http://myproxy:8080"; }},
.. }
{code}
This needs to work both for the main boto3 clients that do task work, but also
during the assume_role process which also uses a boto3 client.
was:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
To set an outbound proxy is then as simple as adding this to your extra_config:
# { .. ,
# "config":{"proxies": {
# "http": "http://myproxy:8080";,
# "https": "http://myproxy:8080"}},
# ..}
> aws_hook: Ability to set outbound proxy
> ---------------------------------------
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws
> Affects Versions: 1.10.6
> Reporter: Bjorn Olsen
> Assignee: Bjorn Olsen
> Priority: Minor
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy
> settings (even if these are set in system wide).
>
> The way to configure a proxy is to pass a botocore.config.Config object to
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS
> Operators, the "config" argument is not used by any operator.
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
> Connection, could allow us to pass kwargs there that build the Config object
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config"
> parameter.
>
> To set an outbound proxy is then as simple as adding this to your
> extra_config:
> {code:java}
> { .. ,
> "config":{ "proxies": {
> "http": "http://myproxy:8080";,
> "https": "http://myproxy:8080"; }},
> .. }
> {code}
>
> This needs to work both for the main boto3 clients that do task work, but
> also during the assume_role process which also uses a boto3 client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
