[
https://issues.apache.org/jira/browse/AIRFLOW-820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524136#comment-16524136
]
Tim Swast commented on AIRFLOW-820:
-----------------------------------
I've documented several of these at
https://airflow.readthedocs.io/en/latest/howto/manage-connections.html#google-cloud-platform
I agree that it would make sense to standardize.
> Standardize GCP related connection id names and default values
> ----------------------------------------------------------------
>
> Key: AIRFLOW-820
> URL: https://issues.apache.org/jira/browse/AIRFLOW-820
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
> Reporter: Feng Lu
> Assignee: Feng Lu
> Priority: Major
>
> A number of Google Cloud Platform (GCP) related operators, such as
> BigQueryCheckOperator or DataFlowJavaOperator, are using different
> connection_id var names and default values. For example,
> BigQueryCheckOperator(.., big_query_conn_id='bigquery_default'..)
> DataFlowJavaOperator(..., gcp_conn_id='google_cloud_default'...)
> DataProcClusterCreateOperator(...,
> google_cloud_conn_id='google_cloud_default',...)
> This makes dag-level default_args problematic, one would have to specify each
> connection_id explicitly in the default_args even though the same GCP
> connection is shared throughout the DAG. We propose to:
> - standardize all connection id names, e.g.,
> big_query_conn_id ---> gcp_conn_id
> google_cloud_conn_id-->gcp_conn_id
> - standardize all default values, e.g.,
> 'bigquery_default' --> 'google_cloud_default'
> Therefore, if the same GCP connection is used, we only need to specify once
> in the DAG default_args, e.g.,
> default_args = {
> ...
> gcp_conn_id='some_gcp_connection_id'
> ...
> }
> Better still, if a connection with the default name 'google_cloud_default'
> has already been created and used by all GCP operators, the gcp_conn_id
> doesn't even need to be specified in DAG default_args.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
