Taragolis commented on PR #25541:
URL: https://github.com/apache/airflow/pull/25541#issuecomment-1205689098
Yeah, I agree that better implement as some abstract class or just raise an
`NotImplementedError` on call specific method
```python
class BaseCopyIntoTable(BaseOperator):
def __init__(self, somearg, some_other_arg, **kwargs):
super.__init__(**kwargs)
...
def from_location(self):
raise NotImplementedError()
def execute(self, context: Context) -> None:
snowflake_hook = SnowflakeHook(...)
sql = f"""
COPY INTO blah.blah FROM self.from_location()
FROM {self.external_location}
Other useful argument for COPY INTO <table>
"""
self.log.info('Executing COPY command...')
snowflake_hook.run(sql)
self.log.info("COPY command completed")
```
And this kind of `BaseCopyIntoTable` could use for multiple different
situations and operators (not required implement all of them in one PR)
**COPY INTO without transformation**
1. Specified named internal/external stage
2. Stage for the specified table
3. Stage for the current user
4. Specified external location for supported cloud providers by providing
storage integration name (provider agnostic way)
5. Specified external location for AWS S3 by providing credentials, not
recommended way and also we need to be sure that credentials not licked to
logs. Obtain credentials by
[AwsHook](https://github.com/apache/airflow/blob/cda40836ca55702c543391367d0829ed231c1b35/airflow/providers/amazon/aws/hooks/base_aws.py#L525-L535)
6. Specified external location for GCS by providing credentials, not
recommended way and also we need to be sure that credentials not licked to logs.
7. Specified external location for Azure Containers by providing
credentials, not recommended way and also we need to be sure that credentials
not licked to logs.
**COPY INTO with transformation**
1. Specified named internal/external stage
2. Stage for the specified table
3. Stage for the current user
Also it would be nice if this abstract operator supports all subset of
snowflake hook parameters which implemented in existed S3ToSnowflakeOperator
https://github.com/apache/airflow/blob/cda40836ca55702c543391367d0829ed231c1b35/airflow/providers/snowflake/transfers/s3_to_snowflake.py#L106-L114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
