[jira] [Updated] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy
[
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bjorn Olsen updated AIRFLOW-6072:
-
Description:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
To set an outbound proxy is then as simple as adding this to your extra_config:
{code:java}
{ .. ,
"config":{ "proxies": {
"http": "http://myproxy:8080;,
"https": "http://myproxy:8080; }},
.. }
{code}
This needs to work both for the main boto3 clients that do task work, but also
during the assume_role process which also uses a boto3 client.
was:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
To set an outbound proxy is then as simple as adding this to your extra_config:
#{ .. ,
#"config":{"proxies": {
#"http": "http://myproxy:8080;,
#"https": "http://myproxy:8080"}},
# ..}
> aws_hook: Ability to set outbound proxy
> ---
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy
> settings (even if these are set in system wide).
>
> The way to configure a proxy is to pass a botocore.config.Config object to
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS
> Operators, the "config" argument is not used by any operator.
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
> Connection, could allow us to pass kwargs there that build the Config object
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config"
> parameter.
>
> To set an outbound proxy is then as simple as adding this to your
> extra_config:
> {code:java}
> { .. ,
> "config":{ "proxies": {
> "http": "http://myproxy:8080;,
> "https": "http://myproxy:8080; }},
> .. }
> {code}
>
> This needs to work both for the main boto3 clients that do task work, but
> also during the assume_role process which also uses a boto3 client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy
[
https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bjorn Olsen updated AIRFLOW-6072:
-
Description:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
To set an outbound proxy is then as simple as adding this to your extra_config:
#{ .. ,
#"config":{"proxies": {
#"http": "http://myproxy:8080;,
#"https": "http://myproxy:8080"}},
# ..}
was:
The boto3 connection used by aws_hook does not respect outbound http_proxy
settings (even if these are set in system wide).
The way to configure a proxy is to pass a botocore.config.Config object to
boto3 when creating a client (according to this SO post).
[https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
While the aws_hook get_client_type() method is used extensively by AWS
Operators, the "config" argument is not used by any operator.
Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
Connection, could allow us to pass kwargs there that build the Config object
automatically by the hook is created.
Otherwise we have to update every AWS Operator to also take a "config"
parameter.
> aws_hook: Ability to set outbound proxy
> ---
>
> Key: AIRFLOW-6072
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6072
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
>
> The boto3 connection used by aws_hook does not respect outbound http_proxy
> settings (even if these are set in system wide).
>
> The way to configure a proxy is to pass a botocore.config.Config object to
> boto3 when creating a client (according to this SO post).
> [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3]
> While the aws_hook get_client_type() method is used extensively by AWS
> Operators, the "config" argument is not used by any operator.
> Adding a check to aws_hook for "config" in the "extra_config" of the Airflow
> Connection, could allow us to pass kwargs there that build the Config object
> automatically by the hook.
> Otherwise we have to update every AWS Operator to also take a "config"
> parameter.
>
> To set an outbound proxy is then as simple as adding this to your
> extra_config:
> #{ .. ,
> #"config":{"proxies": {
> #"http": "http://myproxy:8080;,
> #"https": "http://myproxy:8080"}},
> # ..}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6072) aws_hook: Ability to set outbound proxy
[ https://issues.apache.org/jira/browse/AIRFLOW-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjorn Olsen updated AIRFLOW-6072: - Priority: Minor (was: Major) > aws_hook: Ability to set outbound proxy > --- > > Key: AIRFLOW-6072 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6072 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 1.10.6 >Reporter: Bjorn Olsen >Assignee: Bjorn Olsen >Priority: Minor > > The boto3 connection used by aws_hook does not respect outbound http_proxy > settings (even if these are set in system wide). > > The way to configure a proxy is to pass a botocore.config.Config object to > boto3 when creating a client (according to this SO post). > [https://stackoverflow.com/questions/33480108/how-do-you-use-an-http-https-proxy-with-boto3] > While the aws_hook get_client_type() method is used extensively by AWS > Operators, the "config" argument is not used by any operator. > Adding a check to aws_hook for "config" in the "extra_config" of the Airflow > Connection, could allow us to pass kwargs there that build the Config object > automatically by the hook is created. > Otherwise we have to update every AWS Operator to also take a "config" > parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
