mik-laj commented on issue #8432: Provide GCP credentials in Bash/Python operators URL: https://github.com/apache/airflow/pull/8432#issuecomment-615560449 @potiuk This contradicts the whole idea and the need for this operator. BashOperator and PythonOperaator are very useful because it is universal. Bash and Python are also built by compositions. New applications are installed on the system and can be used by any tool. If we inherit and make customization GCP-specific, we will limit its functionality. It will no longer be a universal operator. You will only be able to use it with one provider. I think this is a similar problem to ``Connection.get_hook``. https://github.com/apache/airflow/blob/master/airflow/models/connection.py#L301 This method is useful because it is universal and can be used regardless of the provider. In the future, if we need to separate the core and providers, we can extend this class with a plugin. A plugin that will add new parameters to the class. Like the get_hook method, it should use the plugin mechanism. From an architectural point of view. Here the use of inheritance will be bad, but we should composition. Inheritance will limit these operators too much. More information: https://en.wikipedia.org/wiki/Composition_over_inheritance I hope that in the future new parameters will be added for other cloud providers, e.g. AWS. ``` cross_platform_task = BashOperator( task_id='gcloud', bash_command=( 'gsutil cp gs//bucket/a.txt a.txt && aws s3 cp test.txt s3://mybucket/test2.txt' ), gcp_conn_id=GCP_PROJECT_ID, aws_conn_id=AWS_PROJECT_ID, ) ``` Then it will still be a universal operator and we will not build a vendor-lock for one providers. However, this is one of the parts that should be resolved by [AIP-8](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
